Apparatus, method and computer-readable medium for audio signal processing

Abstract

The first device includes a memory configured to store instructions and one or more processors configured to receive audio signals from a plurality of microphones. The one or more processors are configured to process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. The one or more processors are also configured to send to a second device data based on the direction-of-arrival information and category or embedded data associated with the direction-of-arrival information.

Description

[Technical Field]

[0001] Cross-reference to related applications

[0002] This patent application claims priority to Provisional Patent Application No. 63 / 203,562 entitled “DIRECTIONAL AUDIO SIGNAL PROCESSING”, filed on July 27, 2021, the contents of which are incorporated herein by reference in their entirety.

[0003] This case involves a comprehensive system related to audio signal processing. [Previous Technology]

[0004] Technological advancements have led to smaller and more powerful computing devices. For example, various portable personal computing devices exist today, including cordless phones (such as mobile phones and smartphones), tablets, and laptops, which are small, lightweight, and easy for users to carry. These devices can transmit voice and data packets via wireless networks. Furthermore, many of these devices incorporate additional functions such as digital still cameras, digital video cameras, digital recorders, and audio file players. Moreover, such devices can process executable instructions, including software applications that can access the Internet (such as web browser applications). Therefore, these devices can include powerful computing capabilities.

[0005] Devices such as mobile phones and smartphones can be paired with headsets to allow users to listen to audio without holding the phone to their ear. One drawback of users wearing headsets is that they may not be aware of their surroundings. As a non-limiting example, if a user is walking across an intersection, they may not hear approaching vehicles. In scenarios where the user's focus is elsewhere (e.g., on their mobile phone or looking away from approaching vehicles), the user may not be able to determine whether a vehicle is approaching or from which direction. [Summary of the Invention]

[0006] According to one embodiment of the present invention, a first device includes a memory configured to store instructions and one or more processors. The one or more processors are configured to receive audio signals from a plurality of microphones. The one or more processors are further configured to process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. The one or more processors are further configured to send to a second device data based on the direction-of-arrival information and category or embedded data associated with the direction-of-arrival information.

[0007] According to another embodiment of the present invention, a method of processing audio signals includes receiving audio signals from a plurality of microphones at one or more processors of a first device. The method further includes processing the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. The method further includes sending to a second device data based on the direction-of-arrival information and category or embedded data associated with the direction-of-arrival information.

[0008] According to another embodiment of the present invention, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause the one or more processors to receive audio signals from a plurality of microphones. When executed by the one or more processors, the instructions also cause the one or more processors to process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. When executed by the one or more processors, the instructions also cause the one or more processors to send to a second device data based on the direction-of-arrival information and category or embedded data associated with the direction-of-arrival information.

[0009] According to another embodiment of the present invention, the first device includes means for receiving audio signals from a plurality of microphones. The first device further includes means for processing the audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals. The first device further includes means for transmitting to a second device data based on the arrival direction information and category or embedded data associated with the arrival direction information.

[0010] Other aspects, advantages and features of this application will become apparent after reading the entire application, including the following sections: description of the drawings, detailed description and claims.

Implementation Method

[0048] A system and method for performing directional audio signal processing are disclosed. A first device (such as a headset) may include a plurality of microphones configured to capture sound from the surrounding environment. Each microphone may have a different orientation and position on the first device, such as to capture sound from different directions. In response to capturing sound, each microphone may generate a corresponding audio signal, which is provided to a directional audio signal processing unit. The directional audio signal processing unit may process the audio signals from the microphones to identify different audio events associated with the sound and location of each audio event. In some embodiments, the audio signals associated with the audio events are processed via one or more classifiers at the first device to identify the audio category of the audio event. In a non-limiting example, if at least one of the plurality of microphones captures a car sound, the directional audio signal processing unit may identify the car sound based on characteristics (e.g., pitch, frequency, etc.) associated with the corresponding audio signal, and may identify the relative direction of the car sound based on the corresponding microphone that captured the sound. In response to identifying a vehicle sound and its corresponding relative direction, the first device can generate data representing the sound and direction, and can provide this data to a second device (such as a mobile phone). In some examples, the data representing the sound may include an audio category or embedding associated with the sound source and direction-of-arrival information. The second device can use this data (e.g., direction information) to perform additional operations. As a non-limiting example, the second device can decide whether to generate a visual alarm or a physical alarm to warn the user of the headset about nearby traffic.

[0049] According to some embodiments, a first device (such as an earphone device) is used to perform distributed audio processing to capture sound using multiple microphones and perform preliminary processing of the audio corresponding to the captured sound. For example, as an illustrative and non-limiting example, the first device may perform arrival direction processing for locating one or more sound sources, acoustic environment processing for detecting the environment or changes in the environment of the first device based on ambient sound, audio event processing for identifying sound corresponding to an audio event, or a combination thereof.

[0050] Because the first device may be relatively limited in terms of processing resources, memory capacity, battery life, etc., the first device may send information about audio processing to a second device (such as a mobile phone) that has greater computing, memory, and power resources. For example, in some embodiments, the first device sends a representation of audio data and a classification of audio events detected in the audio data to the second device, and the second device performs additional processing to verify the classification of the audio events. Depending on some embodiments, the second device uses information provided by the first device (such as directional information and classification associated with the sound events) as additional input to a classifier that processes the audio data. Combining directional information, classification from the first device, or both to perform audio data classification can improve the accuracy, speed, or one or more other aspects of the classifier at the second device.

[0051] This distributed audio processing (such as by providing accurate detection of sound events occurring near the user and enabling the first device to alert the user of the detected events) allows the user of the first device to benefit from the enhanced processing capabilities of the second device. For example, the first device can automatically switch from a playback mode (e.g., playing music or other audio to the user) to a transparent mode, in which the sound corresponding to the detected audio event is played to the user. Other benefits and application examples of the disclosed techniques are described in more detail below with reference to the accompanying drawings.

[0052] Specific aspects of this invention are described below with reference to the accompanying drawings. In the specification, common features are indicated by common element symbols. As used herein, various terms are used only for the purpose of describing particular embodiments and are not intended to limit the embodiments. For example, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some embodiments and plural in others. For illustration, FIG1 illustrates a device 110 including one or more processors ("(multiple) processors" 116 of FIG1), indicating that in some embodiments, device 110 includes a single processor 116, while in other embodiments, device 110 includes multiple processors 116. For ease of reference herein, these features are generally described as "one or more" features and are subsequently referred to in the singular form unless a pattern relating to multiple such features is described.

[0053] It is also understood that the terms "comprise / comprises / comprising" can be used interchangeably with "include / includes / including". Furthermore, it should be understood that the terms "wherein" can be used interchangeably with "where". As used herein, "exemplary" may indicate an example, implementation, and / or manner, and should not be construed as limiting or indicating a preference or preferred implementation. As used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., "first", "second", "third", etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements, while the term "plural" refers to multiple (e.g., two or more) specific elements.

[0054] As used herein, "coupling" can include "communication coupling," "electrical coupling," or "physical coupling," and may (or alternatively) include any combination thereof. Two devices (or elements) may be coupled directly or indirectly (e.g., communication coupling, electrical coupling, or physical coupling) via one or more other devices, elements, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative and non-limiting example, two electrically coupled devices (or elements) may be included in the same or different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some embodiments, two communication-coupled devices (or elements) (e.g., in an electrical communication manner) may directly or indirectly transmit and receive signals (e.g., digital or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" can include two devices coupled (e.g., communication coupling, electrical coupling, or physical coupling) without intermediate elements.

[0055] In this case, terms such as “determine,” “calculate,” “estimate,” “shift,” and “adjust” can be used to describe how one or more operations are performed. It should be noted that these terms should not be construed as restrictive, and similar operations can be performed using other techniques. Furthermore, as referenced herein, “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” can be used interchangeably. For example, “generating,” “calculating,” “estimating,” or “determining” a parameter (or signal) can refer to actively generating, estimating, calculating, or determining a parameter (or signal), or it can refer to using, selecting, or accessing a parameter (or signal) that has already been generated (such as by another element or device).

[0056] Referring to FIG1, a particular illustrative example of a system configured to perform directional processing on multiple audio signals received from multiple microphones is disclosed, and is designated as 100 throughout. System 100 includes a first microphone 102 and a second microphone 104 both coupled to or integrated into device 110. System 100 also includes a third microphone 106 and a fourth microphone 108 coupled to or integrated into device 120. Although two microphones 102, 104 are shown coupled to or integrated into device 110, and two microphones 106, 108 are shown coupled to or integrated into device 120, in other embodiments, device 110, device 120, or both may be coupled to any number of additional microphones. As a non-limiting example, four (4) microphones may be coupled to device 110, and another four (4) microphones may be coupled to device 120. In some embodiments, microphones 102, 104, 106, and 108 are implemented as directional microphones. In other embodiments, one or more (or all) of microphones 102, 104, 106, and 108 are implemented as omnidirectional microphones.

[0057] According to one embodiment, device 110 corresponds to an earphone, and device 120 corresponds to a mobile phone. In some scenarios, device 110 can be paired with device 120 using a wireless connection (e.g., Bluetooth® (a registered trademark of Bluetooth Technology Alliance, Inc., Washington). For example, device 110 can communicate with device 120 using a low-energy protocol (e.g., Bluetooth® Low Energy (BLE) protocol). In other examples, the wireless connection corresponds to signal transmission and reception according to an IEEE 802.11 type (e.g., WiFi) wireless local area network or one or more other radio frequency (RF) communication protocols.

[0058] The first microphone 102 is configured to capture sound 182 from one or more sources 180. In the illustrative example of FIG1, source 180 corresponds to a vehicle, such as a car. Therefore, if device 110 corresponds to headphones, microphones 102, 104 can be used to capture sound 182 from a nearby car. However, it should be understood that vehicles are merely non-limiting examples of sound sources, and the techniques described herein can be implemented with other sound sources. When sound 182 from source 180 is captured, the first microphone 102 is configured to generate an audio signal 170 representing the captured sound 182. Similarly, the second microphone 104 is configured to capture sound 182 from one or more sources 180. When sound 182 from source 180 is captured, the second microphone 104 is configured to generate an audio signal 172 representing the captured sound 182.

[0059] The first microphone 102 and the second microphone 104 can have different positions, different orientations, or both. As a result, microphones 102 and 104 can capture sound 182 at different times, with different phases, or both. For illustration, if the first microphone 102 is closer to the source 180 than the second microphone 104, then the first microphone 102 can capture sound 182 before the second microphone 104 captures it. As described below, if the positions and orientations of microphones 102 and 104 are known, the audio signals 170 and 172 generated by microphones 102 and 104, respectively, can be used to perform directional processing at device 110, device 120, or both. That is, device 110 can use audio signals 170 and 172 to determine the position of the source 180, determine the direction of arrival of sound 182, perform spatial filtering on the audio corresponding to sound 182, etc. As further described below, device 110 can provide device 120 with the results of targeted processing (e.g., data associated with targeted processing) for high-complexity processing, and vice versa.

[0060] Device 110 includes a first input interface 111, a second input interface 112, memory 114, one or more processors 116, and a modem 118. The first input interface 111 is coupled to one or more processors 116 and configured to be coupled to a first microphone 102. The first input interface 111 is configured to receive an audio signal 170 from the first microphone 102 (e.g., a first microphone output) and provide the audio signal 170 to the processor 116 as an audio frame 174. The second input interface 112 is coupled to one or more processors 116 and configured to be coupled to the second microphone 104. The second input interface 112 is configured to receive an audio signal 172 from the second microphone 104 (e.g., a second microphone output) and provide the audio signal 172 to the processor 116 as an audio frame 176. Audio frames 174 and 176 may also be referred to herein as audio data 178.

[0061] One or more processors 116 may optionally include an arrival direction processing unit 132, an audio event processing unit 134, an acoustic environment processing unit 136, a beamforming unit 138, or a combination thereof. According to one embodiment, one or more elements of one or more processors 116 may be implemented using dedicated circuitry. As a non-limiting example, one or more elements of one or more processors 116 may be implemented using a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or the like. According to another embodiment, one or more elements of one or more processors 116 may be implemented by executing instructions 115 stored in memory 114. For example, memory 114 may be a non-transitory computer-readable medium storing instructions 115 that can be executed by one or more processors 116 to perform the operations described herein.

[0062] The direction of arrival processing unit 132 can be configured to process multiple audio signals 170, 172 to generate direction of arrival information 142 corresponding to the source 180 of the sound 182 represented by the audio signals 170, 172. For illustration, the direction of arrival processing unit 132 can select audio frames 174, 176 generated based on audio signals 170, 172 from each microphone 102, 104 that represent similar sounds (such as sound 182 from source 180). For example, the direction of arrival processing unit 132 can process audio frames 174, 176 to compare sound characteristics and ensure that audio frames 174, 176 represent the same instance of sound 182. In an illustrative, non-limiting example of arrival direction processing, in response to determining that audio frames 174 and 176 represent the same instance of sound 182, arrival direction processing unit 132 can compare the timestamps of each audio frame 174 and 176 to determine which microphone 102 and 104 first picked up the corresponding instance of sound 182. If audio frame 174 has an earlier timestamp than audio frame 176, arrival direction processing unit 132 can generate arrival direction information 142 indicating that source 180 is closer to the first microphone 102. If audio frame 176 has an earlier timestamp than audio frame 174, arrival direction processing unit 132 can generate arrival direction information 142 indicating that source 180 is closer to the second microphone 104. Therefore, based on similar timestamps of audio frames 174 and 176, arrival direction processing unit 132 can locate sound 182 and its corresponding source 180. The timestamp from the audio frame of the additional microphone can be used to improve positioning in a similar manner to that described above.

[0063] In some embodiments, instead of the time difference described above, or in addition, one or more other techniques may be used to determine the direction of arrival information 142, such as measuring the phase difference of the sound 182 received at each of the microphones (e.g., microphones 102 and 104) in the microphone array of device 110. In some embodiments, microphones 102, 104, 106, and 108 may be combined with device 120 to operate as a distributed microphone array, and the direction of arrival information 142 is generated based on the characteristics (such as arrival time or phase) of the sound from each of microphones 102, 104, 106, and 108 and based on the relative position and orientation of microphones 102, 104, 106, and 108. In such embodiments, information about sound characteristics (e.g., phase information, time information, or both), captured audio data (e.g., at least a portion of audio signals 170, 172), or a combination thereof may be transmitted between device 110 and device 120 for direction of arrival detection using a distributed microphone array.

[0064] Arrival direction information 142 can be sent to device 120. For example, modem 118 can send data based on arrival direction information 142 to device 120. In some examples, generating arrival direction information 142 at device 110 corresponds to performing a low-complexity processing operation. Device 120 can use arrival direction information 142 to perform a high-complexity processing operation. For example, in some embodiments, device 110 may be a resource-constrained device, such as a device with limited battery life, limited memory capacity, or limited processing power relative to device 120. Performing a high-complexity processing operation at device 120 can offload resource-intensive operations from device 110.

[0065] For illustration, device 120 may optionally include one or more sensors 129. As a non-limiting example, sensor 129 may include non-audio sensors, such as a 360-degree camera, a lidar sensor, etc. Based on the direction of arrival information 142, device 120 may command the 360-degree camera to focus on source 180, command the lidar sensor to measure the distance between the user of device 110, 120 and source 180, etc.

[0066] The audio event processing unit 134 can be configured to process multiple audio signals 170, 172 to perform audio event detection. For illustration, the audio event processing unit 134 can process the sound characteristics of audio frames 174, 176 and compare these sound characteristics with a plurality of audio event models to determine whether an audio event has occurred. For example, the audio event processing unit 134 can access a database (not shown) including models for different audio events (such as car horns, train horns, pedestrian conversations, etc.). In response to a sound characteristic matching (or basic matching) a specific model, the audio event processing unit 134 can generate audio event information 144 indicating that the sound 182 represents an audio event associated with the specific model. As used herein, if the pitch and frequency components of the audio frame are within the threshold values ​​of the pitch and frequency components of a specific sound model, then the sound characteristics of the audio frame can "match" the specific sound model.

[0067] In some embodiments, the audio event processing unit 134 includes one or more classifiers configured to process audio signal data (such as the sound characteristics of audio signals 170, 172, audio frames 174, 176, beamforming data based on audio signals 170, 172, or a combination thereof) to determine an associated category from a plurality of categories supported by the one or more classifiers. In one example, the one or more classifiers operate in combination with the aforementioned plurality of audio event models to determine the category (e.g., classification, such as "dog barking," "glass breaking," "baby crying," etc.) of sounds represented in one or more audio signals and associated with the audio event. For example, the one or more classifiers may include a neural network that has been trained using labeled audio data to distinguish sounds corresponding to various categories and is configured to process the audio signal data to determine a specific category of the sound represented by the audio signal data (or determine the probability that a sound belongs to that category for each category). The category may correspond to or be included in the audio event information 144. An example of a device 110 including one or more classifiers is described in more detail with reference to Figure 6.

[0068] In some embodiments, the audio event processing unit 134 includes one or more encoders configured to process audio signal data (such as audio signals 170, 172, the acoustic characteristics of audio frames 174, 176, beamforming data based on audio signals 170, 172, or a combination thereof) to generate a signature of the sound represented by the audio signal data. For example, the encoder may include one or more neural networks configured to process the audio signal data to generate embeddings corresponding to a specific sound in the audio signal data and associated with an audio event. "Embedding" can specify a relatively low-dimensional space represented by vectors that can be converted from higher-dimensional vectors and can preserve semantic relationships (e.g., an ordered sequence of values ​​or a set of index values). For illustration, the audio signal can be represented using a sequence of relatively large vectors (e.g., representing spectral data and other audio features), which can be processed to generate an embedding represented by smaller vectors. The embedding may include sufficient information to detect a specific sound in the audio signal. A signature (e.g., embedded) may correspond to or be included in audio event information 144. An example of a device 110 including one or more encoders is described in more detail with reference to Figure 7.

[0069] In a non-limiting example, the audio event may correspond to the sound of an approaching vehicle (e.g., source 180). Based on the audio event, the audio event processing unit 134 may generate audio event information 144, and the audio event information 144 may be sent to the device 120. For example, the modem 118 may send data corresponding to the detected event to the device 120. In some examples, generating the audio event information 144 at the device 110 corresponds to performing a low-complexity processing operation. The device 120 may use the audio event information 144 to perform a high-complexity processing operation. To illustrate, based on audio event information 144, device 120 can perform one or more operations, such as processing audio data at a larger, more accurate classifier to verify audio events, editing audio scenes based on sound signatures (e.g., removing sounds corresponding to embeddings included in audio event information 144 or removing sounds not corresponding to embeddings), commanding a 360-degree camera to focus on source 180, commanding a lidar sensor to measure the distance between the user of devices 110 and 120 and source 180, etc.

[0070] The acoustic environment processing unit 136 may be configured to process multiple audio signals 170, 172 to perform acoustic environment detection. For illustration, the acoustic environment processing unit 136 may process the acoustic characteristics of audio frames 174, 176 to determine the acoustic characteristics of the surrounding environment. As a non-limiting example, the acoustic characteristics may include a direct-to-reverberant ratio (DRR) estimate of the surrounding environment. The acoustic environment processing unit 136 may generate environmental information 146 based on the acoustic characteristics of the surrounding environment. For example, if the DRR estimate is relatively high, the environmental information 146 may indicate that the device 110 is in an indoor environment. However, if the DRR estimate is relatively low, the environmental information 146 may indicate that the device 110 is in an outdoor environment. In some embodiments, the acoustic environment processing unit 136 may include or be implemented as one or more classifiers configured to generate outputs indicating audio environment categories that may correspond to or be included in the environmental information 146.

[0071] Environmental information 146 can be sent to device 120. For example, modem 118 can send data to device 120 corresponding to (e.g., identifying) the detected environment. In some examples, generating environmental information 146 at device 110 corresponds to performing low-complexity processing operations. Device 120 can use environmental information 146 to perform high-complexity processing operations. For illustration purposes, as an illustrative and non-limiting example, based on environmental information 146, device 120 can perform one or more operations, such as removing environmental or background noise from one or more audio signals, editing an audio scene based on environmental information 146, or changing the settings of a 360-degree camera to capture outdoor images instead of indoor images.

[0072] Beamforming unit 138 can be configured to process multiple audio signals 170, 172 to perform beamforming. In some examples, beamforming unit 138 performs beamforming based on direction-of-arrival information 142. Alternatively or additionally, in some examples, beamforming unit 138 performs adaptive beamforming, which uses a multi-channel signal processing algorithm to spatially filter the audio signals 170, 172 and determine the location of source 180. Beamforming unit 138 can direct a sensitively enhanced beam to the location of source 180 and suppress audio signals from other locations. In some examples, beamforming unit 138 is configured to (e.g., based on different sound propagation paths from source 180 to each of the different microphones 102, 104, by introducing time or phase delay, adjusting signal amplitude, or both) adjust the processing of audio signal 170 relative to audio signal 172 to (e.g., via constructive interference) emphasize sound arriving from the direction of source 180 and attenuate sound arriving from one or more other directions. In some examples, if beamforming unit 138 determines that the location of source 180 is close to the first microphone 102, beamforming unit 138 may send a command to change the orientation or direction of the first microphone 102 to capture sound 182 and make sound from other directions (e.g., the direction associated with the second microphone 104) null.

[0073] One or more beamformed audio signals 148 (e.g., representations of audio signals 170, 172) may be transmitted to device 120. For example, modem 118 may transmit one or more beamformed audio signals 148 to device 120. In a particular embodiment, a single beamformed audio signal 148 is provided to device 120 for each audio source of interest. In some examples, generating beamformed audio signals 148 at device 110 corresponds to performing low-complexity processing operations. Device 120 may use beamformed audio signals 148 to perform high-complexity processing operations. In an illustrative example, based on beamformed audio signals 148, device 120 may command a 360-degree camera to focus on source 180, command a radar sensor to measure the distance between the user of devices 110, 120 and source 180, etc.

[0074] Optionally, device 110 may send at least a portion of the audio data (e.g., audio signals 170, 172) captured by microphones 102, 104 to device 120 for distributed audio processing, or perform additional processing using greater processing, memory, and power resources available at device 120, in which a portion of the processing described as being performed by device 110 is offloaded to device 120. As an example, in some embodiments, device 110 may send at least a portion of audio signals 170, 172 (e.g., audio data 178) to device 120 for more accurate direction-of-arrival processing, more accurate audio event detection, more accurate environmental detection, or a combination thereof. In some embodiments, beamforming audio signal 148 is not sent, or in addition, device 110 may send at least a portion of audio signals 170, 172 (e.g., audio data 178) to device 120.

[0075] Optionally, device 110 may include or be coupled to a user peripheral device, such as a visual user peripheral device (e.g., a display such as that shown in FIG. 25 or a holographic projection unit such as that shown in FIG. 26, as non-limiting examples), an audio user peripheral device (e.g., a speaker such as that shown with reference to FIG. 3 or a voice user interface such as that shown with reference to FIG. 5, as non-limiting examples), or a tactile user peripheral device (e.g., described with reference to FIG. 22, as a non-limiting example). One or more processors 116 may be configured to provide a user interface output to the user peripheral device indicating at least one of an environmental event or an acoustic event. For illustration, the user interface output may enable the user interface device (e.g., based on audio event information 144, audio event information 145 received from device 120, environmental information 146, environmental information 147 received from device 120, or a combination thereof) to provide notification of detected audio events or environmental conditions.

[0076] The various technical illustrations above illustrate that device 110 (e.g., a low-power device) performs directional context-aware processing. That is, device 110 processes audio signals 170, 172 from multiple microphones 102, 104 to determine the direction from which sound 182 originates. In a particular embodiment, device 110 corresponds to an earphone, and device 120 corresponds to a mobile phone. In this embodiment, the earphone performs directional context-aware processing and can send the resulting data to the mobile phone to perform additional high-complexity processing. In other embodiments, device 110 corresponds to one or more other devices with less computing power compared to device 120 (e.g., a mobile phone, tablet, personal computer, server, vehicle, etc.), such as a headset (e.g., a virtual reality headset, a mixed reality headset, or an augmented reality headset), glasses (e.g., augmented reality glasses or mixed reality glasses), a smartwatch device, a virtual assistant device, or an Internet of Things device.

[0077] As described below, device 120 (e.g., a mobile phone) can also perform directional context-aware processing based on audio signals 170, 172 received from device 110, audio signals 190, 192 from microphones 106, 108, or a combination thereof. Device 120 can provide the results of the directional context-aware processing to device 110 (e.g., an earphone), enabling device 110 to perform additional operations, such as audio scaling operations described in more detail with reference to FIG3.

[0078] The device 120 includes a memory 124, one or more processors 126, and a modem 128. Optionally, the device 120 also includes a first input interface 121, a second input interface 122, and one or more sensors 129.

[0079] In some embodiments, both the first input interface 121 and the second input interface 122 are coupled to one or more processors 126 and configured to be coupled to a third microphone 106 and a fourth microphone 108, respectively. The first input interface 121 is configured to receive an audio signal 190 from the third microphone 106 and to provide the audio signal 190 (such as an audio frame 194) to one or more processors 126. The second input interface 122 is configured to receive an audio signal 192 from the fourth microphone 108 and to provide the audio signal 192 (such as an audio frame 196) to one or more processors 126. The audio signals 190, 192 (e.g., audio frames 194, 196) may be referred to as audio data 198 provided to one or more processors 126.

[0080] One or more processors 126 may optionally include an arrival direction processing unit 152, an audio event processing unit 154, an acoustic environment processing unit 156, a beamforming unit 158, or a combination thereof. According to some embodiments, one or more elements of one or more processors 126 may be implemented using dedicated circuitry. As a non-limiting example, one or more elements of one or more processors 126 may be implemented using an FPGA, an ASIC, or the like. According to another embodiment, one or more elements of one or more processors 126 may be implemented by executing instructions 125 stored in memory 124. For example, memory 124 may be a non-transitory computer-readable medium storing instructions 124, which may be executed by one or more processors 126 to perform the operations described herein.

[0081] The direction of arrival processing unit 152 can be configured to process multiple audio signals (e.g., two or more of audio signals 170, 172, 190, or 192) to generate direction of arrival information 143 corresponding to the source 180 of sound 182 represented by the multiple audio signals. For illustration, the direction of arrival processing unit 152 can be configured to process the multiple audio signals using one or more techniques (e.g., time of arrival, phase difference, etc.) described with reference to the direction of arrival processing unit 132. Compared to the direction of arrival processing unit 132, the direction of arrival processing unit 152 can have more powerful processing capabilities and therefore can produce more accurate results.

[0082] In some embodiments, audio signals 170, 172 are received from device 110, and arrival direction processing unit 152 can process audio signals 170, 172 to determine arrival direction information 143, without processing audio signals 190, 192 at arrival direction processing unit 152. For example, one or more of microphones 106, 108 may be blocked or otherwise unable to produce a useful indication of sound 182, such as when device 120 is a mobile device being carried in a user's pocket or bag.

[0083] In other embodiments, audio signals 190, 192 are received from microphones 106, 108 and processed at the direction of arrival processing unit 152 to determine the direction of arrival information 143, without processing audio signals 170, 172 at the direction of arrival processing unit 152. For example, audio signals 170, 172 may not be transmitted by device 110, or may not be received by device 120. In another example, audio signals 170, 172 may be of low quality (e.g., due to a large amount of noise at microphones 102, 104, such as wind noise), and device 120 may choose to use audio signals 190, 192 and ignore audio signals 170, 172.

[0084] In some embodiments, audio signals 170, 172 are received from device 110 and combined with audio signals 190, 192 at arrival direction processing unit 152 to generate arrival direction information 143. For illustration, device 110 may correspond to an earphone having one or more sensors, such as position or location sensors (e.g., a Global Positioning System (GPS) receiver), one or more inertial measurement units (IMUs) tracking the orientation, movement, or acceleration of device 110 (e.g., head tracker data), or a combination thereof. Device 120 may also include one or more position or location sensors (e.g., a GPS receiver) and an IMU to enable device 120 to combine head tracker data received from device 110 to determine the absolute or relative position and orientation of microphones 102, 104, 106, and 108 operating as a distributed microphone array. Arrival direction information 142, arrival direction information 143, or both, may be relative to the reference frame of device 110, relative to the reference frame of device 120, relative to an absolute reference frame, or a combination thereof, and may be appropriately converted between various reference frames by device 110, device 120, or both.

[0085] Arrival direction information 143 can be sent to device 110. For example, modem 128 can send data to device 110 based on arrival direction information 143. Device 110 can use arrival direction information 143 to perform audio operations, such as audio scaling operations. For example, one or more processors 116 can send commands for capturing (or focusing) audio from the direction of source 180 and sound 182.

[0086] The audio event processing unit 154 can be configured to process multiple audio signals to perform audio event detection and generate audio event information 145 corresponding to one or more detected audio events. For illustration, in embodiments where audio signals 170, 172 are received at device 120, the audio event processing unit 154 can process the sound characteristics of audio signals 170, 172 (e.g., audio frames 174, 176) and compare these sound characteristics with a plurality of audio event models to determine whether an audio event has occurred. In some embodiments where audio signals 190, 192 are received at device 120, the audio event processing unit 154 can process the sound characteristics of audio signals 190, 192 (e.g., audio frames 194, 196) and compare these sound characteristics with a plurality of audio event models to detect audio events. In some embodiments where the beamforming audio signal 148 is received, the audio event processing unit 154 may process the acoustic characteristics of the beamforming audio signal 148 to detect audio events. In some embodiments where the beamforming unit 158 ​​generates the beamforming audio signal 149, the audio event processing unit 154 may process the acoustic characteristics of the beamforming audio signal 149 to detect audio events.

[0087] The audio event processing unit 154 may access a database (not shown) including models for different audio events (such as car horns, train horns, pedestrian conversations, etc.). In response to a sound characteristic matching (or basic matching) of a specific model, the audio event processing unit 154 may generate audio event information 145 indicating that the sound 182 represents an audio event associated with a specific model. In some embodiments, the audio event processing unit 154 includes one or more classifiers configured to determine the category of audio events in a manner similar to that described for audio event processing unit 134. However, compared to audio event processing unit 134, audio event processing unit 154 may perform more complex operations, may support a much larger set of models or audio categories than audio event processing unit 134, and may produce more accurate audio event decisions (or classifications) than audio event processing unit 134.

[0088] In some examples, the audio event processing unit 134 is configured as a relatively low-power detector with relatively high sensitivity, which reduces the probability of audio events going undetected and may also lead to an increase in the number of false alarms (e.g., deciding to detect an audio event when no audio event actually occurs). The audio event processing unit 154 can use information received from the device 110 to provide higher audio event detection accuracy and can verify (e.g., classify) the audio events received from the audio event processing unit 134 by processing the corresponding audio signals (e.g., one or more of audio signals 170, 172, 190, 192, one or more of beamforming audio signals 148, 149, or a combination thereof).

[0089] Audio event information 145 can be sent to device 110. For example, modem 128 can send data corresponding to a detected event to device 110. Device 110 can use audio event information 145 to perform audio operations, such as audio scaling operations. For example, one or more processors 116 can send commands to capture (or focus) the sound from the audio event. In another example, audio event information 145 can cause one or more processors 116 to ignore (e.g., defocus) or attenuate or remove the sound from the audio event. For example, audio event processing unit 154 can determine that the audio event corresponds to the buzzing of a fly near device 110, and audio event information 145 can instruct device 110 to ignore the buzzing or to guide a zero beam in the direction of the source of the buzzing. In an implementation where device 110 selects whether to replay ambient sounds to the user of device 110, such as when device 110 is configured to enter "transparent" mode so that the user can hear external sounds in a specific environment, audio event information 145 can indicate to device 110 whether sound 182 should trigger device 110 to switch to transparent mode.

[0090] The acoustic environment processing unit 156 can be configured to process multiple audio signals 170, 172, multiple audio signals 190, 192, or combinations thereof to perform acoustic environment detection. For illustration, the acoustic environment processing unit 156 can process the acoustic characteristics of audio frames 174, 176, audio frames 194, 196, or both to determine the acoustic characteristics of the surrounding environment. In some embodiments, the acoustic environment processing unit 156 operates in a similar manner to the acoustic environment processing unit 136. However, compared to the acoustic environment processing unit 136, the acoustic environment processing unit 156 can perform more complex operations, can support a much larger set of models or audio environment categories than the acoustic environment processing unit 136, and can produce more accurate acoustic environment decisions (or classifications) than the acoustic environment processing unit 136.

[0091] In some examples, the acoustic environment processing unit 136 is a relatively low-power detector that is configured to have relatively high sensitivity to environmental changes (e.g., as a non-limiting example, detecting changes in background sound characteristics when device 110 moves from an indoor environment to an outdoor environment or from an outdoor environment to a vehicle) but may have relatively low accuracy in determining the environment itself. The acoustic environment processing unit 156 can use the information received from device 110 to provide higher acoustic environment detection accuracy and can verify the environmental information 146 received from the acoustic environment processing unit 136 (e.g., classification) by processing corresponding audio signals (e.g., one or more of audio signals 170, 172, 190, 192, one or more of beamforming audio signals 148, 149, or a combination thereof).

[0092] The acoustic environment processing unit 156 can generate environmental information 147 based on the acoustic characteristics of the surrounding environment. The environmental information 147 can be sent to the device 110. For example, the modem 128 can send data corresponding to the detected environment to the device 110. The device 110 can use the environmental information 147 to perform additional audio operations.

[0093] Beamforming unit 158 ​​can be configured to process multiple audio signals 170, 172 to perform adaptive beamforming. For illustration, in some examples, beamforming unit 158 ​​uses a multi-channel signal processing algorithm to spatially filter audio signals 170, 172 to direct a more sensitive beam towards the location of source 180, and suppresses audio signals from other locations in a manner similar to that described for beamforming unit 138. In another example, beamforming unit 158 ​​uses a multi-channel signal processing algorithm to spatially filter audio signals 190, 192 to direct a more sensitive beam towards the location of source 180. In another example where device 120 receives audio signals 170, 172 from device 110 and also receives audio signals 190, 192, beamforming unit 158 ​​can perform spatial filtering based on all audio signals 170, 172, 190, and 192. In some implementations, beamforming unit 158 ​​generates a single beamformed audio signal for each sound source detected in the audio signal. For example, if a single sound source is detected, a single beamformed audio signal 149 is generated pointing to that sound source. In another example, if multiple sound sources are detected, multiple beamformed audio signals 149 can be generated, each of the multiple beamformed audio signals 149 pointing to a corresponding sound source.

[0094] The resulting beamformed audio signal 149 can be sent to device 110. For example, modem 128 can send one or more beamformed audio signals 149 to device 110. Device 110 can use the beamformed audio signal 149 to replay the improved audio.

[0095] Although various elements of devices 110 and 120 have been shown and described above, it should be understood that in other embodiments, one or more elements may be omitted or bypassed. It should also be understood that various combinations of elements of devices 110, 120, or both can achieve interoperability that enhances the performance of devices 110, 120, or both, as described in the non-limiting examples listed below.

[0096] In a particular embodiment, device 110 includes an audio event processing unit 134, and omits the direction of arrival processing unit 132, the acoustic environment processing unit 136, and the beamforming unit 138 (or disables or bypasses their operation). In this embodiment, as described above, audio event information 144 may be provided to device 120 and used in conjunction with processing that uses audio signals 170, 172, audio signals 190, 192, or a combination of audio signals 170, 172, 190, 192 at device 120.

[0097] In another specific embodiment, device 110 includes an audio event processing unit 134 and an arrival direction processing unit 132, omitting the acoustic environment processing unit 136 and beamforming unit 138 (or disabling or bypassing their operation). In this embodiment, arrival direction information 142 and audio event information 144 are generated at device 110 and can be provided to device 120 for the purposes described above. Arrival direction information 142 can be used (e.g., via increased accuracy, reduced latency, or both) to enhance audio event detection that can be performed at audio event processing unit 134, audio event processing unit 154, or both. For example, arrival direction information 142 can be provided as input to audio event processing unit 134, and audio event processing unit 134 can compare arrival direction information 142 with directions associated with one or more previously detected audio events or sound sources. In another example, audio event processing unit 134 can use arrival direction information 142 to increase or decrease the likelihood of detecting a particular audio event. To illustrate, since sounds originating from above the user are more likely to come from birds or airplanes than cars, a weighting factor can be applied, as an illustrative and non-limiting example, to reduce the probability that an overhead sound is identified as matching a car-based audio event. Additionally or alternatively, the direction of arrival information 142 can be used to enhance the performance of the audio event processing unit 154 in a manner similar to that described for the audio event processing unit 134.

[0098] As further explained with reference to FIG. 9, the performance of the audio event processing unit 154 can be enhanced by providing audio event information 144 (e.g., the audio category detected by the audio event processing unit 134) as input to the audio event processing unit 154. For example, the audio event information 144 can be used as a starting point for searching the event model database, or as input that can influence the classification operation performed by a neural network-based audio event classifier. Therefore, by using the direction of arrival information 142 at the audio event processing unit 134 to improve the accuracy of the audio event information 144, the improved accuracy of the audio event information 144 can also improve the performance of the audio event processing unit 154.

[0099] In some embodiments of the device 110, which also includes an acoustic environment processing unit 136, environmental information 146 can be used to improve the performance of audio event processing unit 134, audio event processing unit 154, or both. For example, because some audio events (e.g., car horns) are more likely to occur in some environments (e.g., on a busy street or in a vehicle) than in others (e.g., in an office), audio event processing unit 134 can adjust its operation based on the environment. For example, audio event processing unit 134 can prioritize searching for sound event models that are more likely to occur in a particular environment, which can result in increased accuracy, reduced latency, or both. As another example, audio event processing unit 134 can adjust the weighting factors for one or more sound event models based on the environment to increase or decrease the likelihood that sound 182 is determined to match those sound event models. In some embodiments, environmental information 146 can be sent to device 120 and used to improve the performance of audio event processing unit 154 in a similar manner.

[0100] In some embodiments of the device 110 that include a beamforming unit 138, a beamforming audio signal 148 can be used to improve the operation of an audio event processing unit 134, an audio event processing unit 154, or both. For example, the beamforming audio signal 148 can be directed to a source 180 of sound 182, thus amplifying sound 182, attenuating or removing sound or ambient noise from other sources, or combinations thereof. As a result, in embodiments where the audio event processing unit 134 operates the beamforming audio signal 148, the beamforming audio signal 148 can provide an improved representation of sound 182 compared to audio signals 170, 172, enabling the audio event processing unit 134 to determine audio event information 144 more accurately (e.g., by reducing the likelihood of misclassification of sound 182). Similarly, in an embodiment where beamforming audio signal 148 is sent to device 120 and audio event processing unit 154 operates on beamforming audio signal 148, beamforming audio signal 148 can achieve improved performance of audio event processing unit 154.

[0101] In a particular embodiment, device 120 includes an audio event processing unit 154, and omits the direction of arrival processing unit 152, the acoustic environment processing unit 156, and the beamforming unit 158 ​​(or disables or bypasses their operation). In this embodiment, as described above, the audio event processing unit 154 can operate using audio signals 170, 172, beamformed audio signal 148, audio signals 190, 192, or combinations thereof.

[0102] In another specific embodiment, device 120 includes an audio event processing unit 154 and an arrival direction processing unit 152, omitting the acoustic environment processing unit 156 and the beamforming unit 158 ​​(or disabling or bypassing their operation). In this embodiment, arrival direction information 143 and audio event information 145 are generated at device 120 and can be provided to device 110 for use as described above. Arrival direction information 143 can be used (e.g., via increased accuracy, reduced latency, or both) to enhance audio event detection that can be performed at audio event processing unit 154 in a manner similar to that described for arrival direction information 142.

[0103] In some embodiments where the device 120 also includes an acoustic environment processing unit 156, environmental information 147 can be used to improve the performance of audio event processing unit 134, audio event processing unit 154, or both, in a manner similar to that described for environmental information 146. In some embodiments where the device 120 includes a beamforming unit 158, beamformed audio signals generated by beamforming unit 158 ​​can be used to improve the operation of audio event processing unit 154 in a manner similar to that described for beamformed audio signals 148.

[0104] The techniques described with reference to FIG1 enable each device 110, 120 to perform directional context-aware processing based on audio signals 170, 172 generated by microphones 102, 104, audio signals 190, 192 generated by microphones 106, 108, or combinations thereof. As a result, each device 110, 120 is able to detect the context for different use cases and determine the characteristics associated with the surrounding environment. As a non-limiting example, these techniques enable each device 110, 120 to distinguish one or more mobile sound sources (e.g., whistles, birds, etc.), one or more stationary sound sources (e.g., televisions, speakers, etc.), or combinations thereof.

[0105] It should be understood that the techniques described with respect to FIG1 can enable multi-channel or single-channel audio context detection to distinguish different sounds based on direction of arrival. According to one embodiment, microphones 102, 104, 106, and 108 can be included in a microphone array having microphones located at different locations within a building (such as a house). In a scenario where someone falls to the floor, if the microphones of the microphone array are connected to a mobile device (such as device 120), using the techniques described herein, the mobile device can use the direction of arrival information to determine where the sound is coming from, determine the context of the sound, and perform appropriate actions (e.g., notify caregivers).

[0106] Referring to FIG2, another specific illustrative example of a system configured to perform directional processing on multiple audio signals received from multiple microphones is disclosed, and is generally designated as 200. System 200 includes one or more processors 202. The one or more processors 202 may be integrated into device 110 or device 120. For example, the one or more processors 202 may correspond to one or more processors 116 or one or more processors 126.

[0107] One or more processors 202 may optionally include an audio input 204 configured to receive audio data 278 (such as audio data 178 in FIG. 1) and output audio frames 274, 276. One or more processors 202 include a first processing domain 210 and a second processing domain 220. The first processing domain 210 may correspond to a low-power domain operating in a low-power state, such as an "always-on" power domain. The first processing domain 210 may remain active to process audio frames 274 and 276. In some embodiments, audio frames 274 and 276 correspond to audio frames 174 and 176, respectively. In another embodiment, audio frames 274 and 276 correspond to audio frames 194 and 196, respectively. The second processing domain 220 may correspond to a high-power domain that transitions between an idle state and a high-power state.

[0108] The first processing domain 210 includes an audio preprocessing unit 230. Compared to one or more elements in the second processing domain 220, the audio preprocessing unit 230 can consume relatively low power. The audio preprocessing unit 230 can process audio frames 274, 276 to determine whether any audio activity exists. According to some embodiments, the audio preprocessing unit 230 can receive and process audio frames from a single microphone to save additional power. For example, in some embodiments, audio frame 276 may not be provided to the first processing domain 210, and the audio preprocessing unit 230 can determine whether audio activity exists in audio frame 274.

[0109] If the audio preprocessing unit 230 determines that there is audio activity in audio frame 274 or both audio frames 274, 276, the audio preprocessing unit 230 may generate a start signal 252 to switch the second processing domain 220 from an idle state to a high-power state. According to some embodiments, the audio preprocessing unit 230 may determine preliminary direction information 250 regarding audio activity and provide the preliminary direction information 250 to the second processing domain 220. For example, if there is audio activity in audio frame 274 and a small amount of audio activity or no audio activity in audio frame 276, the preliminary direction information 250 may indicate that the sound 182 originates from the vicinity of the microphone that captured the audio signal corresponding to audio frame 274.

[0110] The second processing domain 220 includes an arrival direction processing unit 232, an audio event processing unit 234, an acoustic environment processing unit 236, a beamforming unit 238, or a combination thereof. The arrival direction processing unit 232 may correspond to the arrival direction processing unit 132 or 152 of FIG. 1, and may operate in a substantially similar manner. The audio event processing unit 234 may correspond to the audio event processing unit 134 or 154 of FIG. 1, and may operate in a substantially similar manner. The acoustic environment processing unit 236 may correspond to the acoustic environment processing unit 136 or 156 of FIG. 1, and may operate in a substantially similar manner. The beamforming unit 238 may correspond to the beamforming unit 138 or 158 of FIG. 1, and may operate in a substantially similar manner.

[0111] Therefore, the second processing domain 220 can operate in different modes. For example, the second processing domain 220 can be used to activate different sensors, such as sensor 129 in FIG1. ​​In addition, the second processing domain 220 can be used to perform direction of arrival processing and calculation, beamforming, DRR operation, indoor / outdoor detection, source distance determination, etc.

[0112] The system 200 enables the first processing domain 210 to selectively activate the second processing domain 220 in response to the detection of audio activity. As a result, when no audio activity is detected by using low-power processing, battery power can be saved in devices such as headsets or mobile phones by switching the second processing domain 220 (e.g., a high-power processing domain) to an idle state.

[0113] Referring to Figure 3, another specific illustrative example of a system configured to perform directional processing on multiple audio signals received from multiple microphones is disclosed, and is generally designated as 300. System 300 includes an earpiece 310 and a mobile phone 320. The earpiece 310 may correspond to device 110, and the mobile phone 320 may correspond to device 120.

[0114] The earphone 310 includes an audio processing unit 330, an audio scaling unit 332, an optional user prompt generation unit 334, or a combination thereof. The audio processing unit 330 includes an arrival direction processing unit 132 and an audio event processing unit 134. As shown in FIG1, the arrival direction processing unit 132 can generate arrival direction information 142 indicating the location (e.g., direction of orientation) of the source 180 of the sound 182. The arrival direction information 142 is provided to the audio scaling unit 332 and the user prompt generation unit 334. As shown in FIG1, the audio event processing unit 134 can generate audio event information 144 indicating that the sound 182 is related to a vehicle sound. The audio event information 144 is provided to the user prompt generation unit 334.

[0115] The audio scaling unit 332 can also receive arrival direction information 143 from the mobile phone 320. The audio scaling unit 332 can be configured to adjust the beamforming algorithm of the beamforming unit 138 based on the arrival direction information 142 or the arrival direction information 143. As a result, the audio scaling unit 332 can focus the microphones 102, 104 on the sound of interest (e.g., sound 182) and attenuate sounds from other directions. The headset 310 can therefore generate a beamformed audio signal 148 focused on the sound 182 from the source 180 and provide the beamformed audio signal 148 to the speaker 336 for playback. In some implementations, beamforming audio signals 148 are reproduced at multiple speakers 336 (e.g., a left speaker for the user's left ear and a right speaker for the user's right ear) in a manner that maintains the directionality of the source 180 of sound 182, so that the user perceives the focused sound 182 as originating from the direction of the source 180 (or, if distance information is determined, from the location of the source 180).

[0116] The user prompt generation unit 334 can generate a user alarm 350 that is provided to the speaker 336 for replay. For example, the user alarm 350 may be an audio message indicating that a vehicle (e.g., source 180) is approaching. The user prompt generation unit 334 may also generate one or more user alarms 352 that are provided to the mobile phone 320. The user alarm 350 may include text indicating that a vehicle is approaching, vibrations programmed to indicate that a vehicle is approaching, etc.

[0117] Therefore, the system 300 of FIG3 enables the headphones 310 to focus (e.g., audio scaling) on ​​the sound of interest 182 and can generate user alarms 350, 352. For illustration, in a scenario where the user is wearing the headphones 310, the system 300 can alert the user to surrounding events (such as an approaching vehicle) that the user might otherwise not be aware of.

[0118] Referring to Figure 4, another specific illustrative example of a system configured to perform directional processing on multiple audio signals received from multiple microphones is disclosed, and is generally designated as 400. System 400 includes a headset 410 and a mobile phone 420. The headset 410 may correspond to device 110, and the mobile phone 420 may correspond to device 120.

[0119] The earphone 410 includes an audio processing unit 430, and optionally includes an audio scaling unit 432, a noise reduction unit 434, one or more speakers 436, or a combination thereof. The audio processing unit 430 includes a direction of arrival processing unit 132 and an audio event processing unit 134. As shown in FIG1, the direction of arrival processing unit 132 can generate direction of arrival information indicating the approximate location of the source 180 of the sound 182. The direction of arrival processing unit 132 can also generate direction of arrival information indicating the approximate location of the source 184 of the sound 186. As shown in FIG1, the audio event processing unit 134 can generate audio event information indicating that the sound 182 is related to vehicle sounds. The audio event processing unit 134 can also generate audio event information indicating that the sound 186 is related to human speech.

[0120] The audio processing unit 430 may be configured to generate first audio information 440, which indicates arrival direction information associated with sound 182 (e.g., a first output of arrival direction processing unit 132) and indicates that sound 182 is associated with a vehicle (e.g., a first output of audio event processing unit 134). The audio processing unit 430 may also be configured to generate second audio information 442, which indicates arrival direction information associated with sound 186 (e.g., a second output of arrival direction processing unit 132) and indicates that sound 186 is associated with human speech (e.g., a second output of audio event processing unit 134). Optionally, the headset 410 may send audio signal data, such as one or more portions of audio signals 170, 172 corresponding to sounds 182, 186, to the mobile phone 420. The audio signal data may be included in the audio information 440, 442, or may be separate from the audio information 440, 442.

[0121] The mobile phone 420 includes a single-microphone audio context detection unit 450, an audio adjustment unit 452, and a mode controller 454. First audio information 440 and second audio information 442 are provided to the audio adjustment unit 452. According to some embodiments, the single-microphone audio context detection unit 450 may provide the audio adjustment unit 452 with additional context information 496, such as arrival direction information 143 generated by the arrival direction processing unit 152 of FIG. 1, audio event information 145 generated by the audio event processing unit 154, environmental information 147 generated by the acoustic environment processing unit 156, or combinations thereof. For example, the single-microphone audio context detection unit 450 may process audio signal data received from the headset 410 (e.g., one or more portions of audio signals 170, 172), audio signal data received from one or more microphones of the mobile phone 420 (e.g., audio signals 190, 192), or combinations thereof.

[0122] The audio adjustment unit 452 can be configured to generate an audio scaling angle 460 and a noise reduction parameter 462 based on the audio information 440, 442 from the audio processing unit 430. That is, based on context information 496 from the single-microphone audio context detection unit 450, the audio adjustment unit 452 can determine the audio scaling angle 460 to focus for beamforming purposes, and can determine the noise reduction parameter 462 to reduce noise from other directions. Therefore, based on the context information 496, if the audio adjustment unit 452 decides to preferentially focus on the sound 182, the audio scaling angle 460 can indicate the angle associated with the source 180, and the noise reduction parameter 462 can include parameters for reducing noise from the source 184. The audio scaling angle 460 is provided to the audio scaling unit 432, and the noise reduction parameter 462 is provided to the noise reduction unit 434.

[0123] The audio adjustment unit 452 can also be configured to generate a mode signal 464 provided to the mode controller 454. The mode signal 464 can indicate whether a vibration alarm should be generated for the user of the mobile phone 420, whether a text alarm should be generated for the user of the mobile phone 420, whether a voice alarm should be generated for the user of the mobile phone 420, etc.

[0124] The audio scaling unit 432 can be configured to adjust the beamforming algorithm of the beamforming unit (such as beamforming unit 138 of FIG. 1) based on the audio scaling angle 460. As a result, the audio scaling unit 432 can focus the microphones 102, 104 on the sound of interest (e.g., sound 182). Based on the noise reduction parameter 462, the noise reduction unit 434 can be configured to generate a noise reduction signal 490 to attenuate sound 186 from other directions. The beamformed audio signal 148 and the noise reduction signal 490 can be provided to one or more speakers 436 for playback.

[0125] The system 400 in Figure 4 can analyze detected sound events and their corresponding directions of arrival to improve auditory experience. Based on contextual information 496, the system 400 can determine which sound the user is particularly interested in. For example, if the user is crossing the street, the system 400 can determine that the sound of vehicles 182 is more important than the sound of people talking 186. As a result, the system 400 can focus on the important sound 182 and suppress other sounds.

[0126] Although the headphones 410 are described as providing focusing on sound 182 and suppression of other sounds, it should be noted that each of the focusing on sound 182 provided by the audio scaling unit 432 and the suppression of other sounds provided by the noise reduction unit 434 provides the user of the headphones 410 with an enhanced perception of sound 182. For example, in an embodiment where the headphones 410 includes the audio scaling unit 432 but omits the noise reduction unit 434 (or bypasses the operation of the noise reduction unit 434), sound 182 is enhanced via audio scaling even in the absence of a noise reduction signal 490. As another example, in an embodiment where the headphones 410 includes the noise reduction unit 434 but omits the audio scaling unit 432 (or bypasses the operation of the audio scaling unit 432), sound 182 is enhanced relative to other sounds via noise reduction applied to other sounds.

[0127] Referring to FIG5, another specific illustrative example of a system configured to perform directional processing on multiple audio signals received from multiple microphones is disclosed, and is generally designated as 500. System 500 includes a spatial filtering processing unit 502, an audio event processing unit 504, an application programming interface 506, and a voice user interface 508. According to one embodiment, system 500 may be integrated into device 110 or device 120.

[0128] The spatial filtering processing unit 502 can be configured to perform one or more spatial filtering operations on audio frames (shown as audio frames 574 and 576) associated with the received audio signal. In some embodiments, audio frames 574 and 576 correspond to audio frames 174 and 176, respectively. In another embodiment, audio frames 574 and 576 correspond to audio frames 194 and 196, respectively. In a non-limiting example, the spatial filtering processing unit 502 can perform adaptive beamforming, audio scaling, beamforming, zero beamforming, or combinations thereof on audio frames 574 and 576.

[0129] Based on spatial filtering operations, the spatial filtering processing unit 502 can generate a plurality of outputs 510, 512, 514 and corresponding direction-of-arrival information 542 for each output 510, 512, 514. In the illustrative example of FIG5, the spatial filtering processing unit 502 can generate a speech content output 510 based on audio frames 574, 576 and two other outputs 512, 514 (e.g., audio from two other detected audio sources). The outputs 510, 512, 514 are provided to the audio event processing unit 504, and the direction-of-arrival information 542 for each output 510, 512, 514 is provided to the application programming interface 506.

[0130] The audio event processing unit 504 is configured to process each output 510, 512, 514 to determine the audio event information 544 associated with each output 510, 512, 514. For example, the audio event processing unit 504 may instruct output 510 to be associated with voice content, output 512 to be associated with non-voice content, and output 514 to be associated with non-voice content. The audio event processing unit 504 provides the voice content output 510 to the voice user interface 508 for user replay and provides the audio event information 544 to the application programming interface 506.

[0131] The application programming interface 506 can be configured to provide arrival direction information 542 and audio event information 544 to other applications or devices for further specialized processing, as shown in Figures 1-4.

[0132] FIG6 illustrates an embodiment 600 of device 110. One or more processors 116 are configured to receive audio signals (shown as audio signals 170, 172) from multiple microphones. One or more processors 116 are also configured to send data to a second device based on a category 612 of sounds represented by one or more of audio signals 170, 172 and associated with an audio event. For example, one or more processors 116 send an indication 616 of category 612 to the second device (e.g., device 120). In an illustrative example, one or more processors 116 are integrated into a headset device, and the second device corresponds to a mobile phone. In another illustrative example, one or more processors 116 are integrated into a vehicle.

[0133] One or more processors 116 are configured to process signal data 602 at one or more classifiers 610 to determine category 612 from a plurality of supported categories 614 supported by one or more classifiers 610. Signal data 602 corresponds to audio signals 170, 172. For example, in some embodiments, one or more processors are configured (e.g., at beamforming unit 138) to perform beamforming operations on audio signals 170, 172 to generate signal data 602, which may correspond to beamformed audio signal 148. Alternatively or additionally, one or more processors 116 are configured to determine one or more features of audio signals 170, 172 to be included in signal data 602. Alternatively or additionally, signal data 602 includes audio signals 170, 172.

[0134] Depending on some embodiments, one or more classifiers 610 include one or more neural networks configured to process signal data 602 and generate outputs indicating that category 612 is more closely associated with the audio event than the remaining categories among a plurality of supported categories 614 (e.g., monostable output). Category 612 is sent to a second device via indication 616. In some examples, indication 616 includes the bit configuration, quantity, or other indication of category 612. In other examples, indication 616 includes a text name, label, or other descriptor that enables category 612 to be recognized by the second device. In some embodiments, one or more classifiers 610 correspond to (or are included in) the audio event processing unit 134 of FIG1, and indication 616 corresponds to (or is included in) audio event information 144.

[0135] Optionally, one or more processors 116 are also configured to process image data at one or more classifiers 610 to determine category 612. For example, device 110 may optionally include one or more cameras configured to generate image data, or may receive image data from another device (e.g., via a modem). Category 612 may correspond to an object (e.g., a sound source) represented in the image data and associated with an audio event. For example, in some embodiments, one or more processors 116 may generate direction of arrival information 142 based on audio signals 170, 172 (or receive direction of arrival information 143 from a second device), and use the direction of arrival information 142 or 143 to locate the object corresponding to the sound source in the image data. In embodiments where one or more classifiers 610 process image data in addition to processing audio data, the image data may be included in signal data 602 or provided as a separate input to one or more classifiers 610.

[0136] In some implementations, the plurality of supported categories 614 includes an "unknown" category, which means that the audio event failed to correspond to any other supported category 614 within a confidence threshold. In one example, one or more classifiers 610 calculate the probability that the audio event corresponds to that particular category for each of the plurality of supported categories 614. If none of the calculated probabilities exceed the threshold amount, one or more classifiers 610 designate category 612 as the "unknown" category.

[0137] In some embodiments, one or more processors 116 are configured to process audio signals 170, 172 to generate direction-of-arrival information corresponding to one or more sources of sound represented in one or more of the audio signals, and a category 612 is associated with the direction-of-arrival information. For example, the direction-of-arrival information and category 612 correspond to the same sound in audio signals 170, 172. For illustration, one or more processors 116 may optionally include the direction-of-arrival processing unit 132 of FIG1. ​​One or more processors 116 may be configured to send data based on the direction-of-arrival information to a second device. In one example, the data based on the direction-of-arrival information includes a report indicating at least one detected event and the direction of the detected event.

[0138] According to various embodiments, device 110 may optionally include one or more additional elements or patterns previously described with reference to FIG. 1. For example, one or more processors may be configured to perform spatial processing on audio signals based on direction-of-arrival information to generate one or more beamforming audio signals, and may transmit one or more beamforming audio signals to a second device. For illustration, one or more processors 116 may optionally include the beamforming unit 138 of FIG. 1. In another example, one or more processors 116 may be configured to generate environmental data corresponding to the detected environment based on acoustic environment detection operations. For illustration, one or more processors 116 may optionally include the acoustic environment processing unit 136 of FIG. 1.

[0139] In another example, one or more processors 116 may be configured to send representations of audio signals 170, 172 to a second device. In some embodiments, the representations of audio signals 170, 172 correspond to one or more beamforming audio signals, such as beamforming audio signal 148. In another example, one or more processors 116 may be configured to receive directional information associated with the audio signals from the second device and perform audio scaling operations based on the directional information, as described with reference to Figures 3 and 4.

[0140] By sending an instruction 616 corresponding to a category 612 of the sound represented by audio signals 170, 172, device 110 provides information that can be used by a second device to improve the accuracy of audio event processing at the second device, as further described with reference to FIG9.

[0141] FIG7 illustrates an embodiment 700 of the device 110. Compared to embodiment 600, embodiment 700 includes one or more encoders 710 and omits one or more classifiers 610. Signal data 602 is processed by one or more encoders 710 to produce an embedding 712 corresponding to sounds represented by one or more of audio signals 170, 172 and associated with audio events. One or more processors 116 are also configured to send data based on the embedding 712 to a second device. In one example, one or more processors 116 send an instruction 716 of the embedding 712 to the second device.

[0142] According to some embodiments, one or more encoders 710 include one or more neural networks with embeddings 712 configured to process signal data 602 to generate sound. The embedding 712 represents a "signature" of the sound, which includes sufficient information about various characteristics of the sound to enable the sound to be detected in other audio signals, but may not include sufficient information to reproduce the sound solely from the embedding 712. According to some embodiments, the embedding 712 may correspond to a user's voice, a specific sound from the environment (such as a dog barking), and the embedding 712 may be used to detect and amplify or extract other instances of sound that may occur in other audio data, as further described with reference to FIG11. In some embodiments, one or more encoders 710 correspond to (or are included in) the audio event processing unit 134 of FIG1, and indication 716 corresponds to (or is included in) audio event information 144.

[0143] In some embodiments, one or more processors 116 are configured to process audio signals 170, 172 to generate direction-of-arrival information corresponding to one or more sources of sound represented in the audio signals, and an embedding 712 is associated with the direction-of-arrival information. In one example, the direction-of-arrival information and the embedding 712 correspond to the same sound in the audio signals 170, 172. For illustration, one or more processors 116 may optionally include the direction-of-arrival processing unit 132 of FIG1. ​​One or more processors 116 may be configured to send data based on the direction-of-arrival information to a second device.

[0144] Optionally, one or more processors 116 are also configured to process image data at one or more encoders 710 to generate embedding 712. For example, device 110 may optionally include one or more cameras configured to generate image data, or may receive image data from another device (e.g., via a modem). Embedding 712 may correspond to an object (e.g., a sound source) represented in image data and associated with an audio event. For example, in some embodiments, one or more processors 116 may generate direction of arrival information 142 based on audio signals 170, 172 (or receive direction of arrival information 143 from a second device), and use the direction of arrival information 142 or 143 to locate an object corresponding to a sound source in the image data. In embodiments where one or more encoders 710 process image data in addition to processing audio data, the image data may be included in signal data 602 or provided as a separate input to one or more encoders 710.

[0145] FIG8 illustrates an embodiment 800 of device 110, which includes one or more classifiers 610 of FIG6 and one or more encoders 710 of FIG7. Signal data 602 (or one or more portions of signal data 602) is processed by one or more classifiers 610 to determine category 612, and signal data 602 (or one or more portions of signal data 602) is processed by one or more encoders 710 to generate embedding 712. One or more processors 116 are also configured to send data based on category 612, embedding 712, or both to a second device. For example, an indication 616 of category 612, an indication 716 of embedding 712, or both may correspond to or be included in the audio event processing unit 134 of device 120 of FIG1.

[0146] FIG9 illustrates an embodiment 900 of a device 120 (e.g., a second device) including one or more processors 126. The one or more processors 126 include an audio event processing unit 154 and are configured to receive an indication 902 from a first device (e.g., device 110) of an audio category corresponding to an audio event. In some examples, indication 902 corresponds to indication 616 of FIG6 or FIG8, which indicates a category 612 detected at one or more classifiers 610 of device 110. In some embodiments, one or more processors 126 are coupled to memory (e.g., memory 124) and integrated into a mobile phone, and the first device corresponds to a headset device. In another embodiment, memory and one or more processors 126 are integrated into a vehicle.

[0147] Optionally, one or more processors 126 include one or more classifiers 920, which may correspond to or be included in the audio event processing unit 154. Depending on the configuration, one or more classifiers 920 may be more powerful and accurate than the classifier(s) in the first device that generates indication 902, as described with reference to the audio event processing unit 154 of FIG1. ​​One or more processors 126 may be configured to also receive audio data 904 representing sound associated with an audio event. In some embodiments, as an illustrative and non-limiting example, the audio data 904 may correspond to audio signals 170, 172 from the first device, beamformed audio signals 148, 190, 192, or combinations thereof from the first device. One or more processors 126 may be configured to process audio data 904 at one or more classifiers 920 to verify that indication 902 is correct (e.g., by comparing indication 902 with a category 922 determined by one or more classifiers 920). Category 922 may be selected from a plurality of supported categories 924 as the audio category that best corresponds to the audio event detected in audio data 904.

[0148] In some embodiments, verifying indication 902 or verifying the category indicated by indication 902 includes determining whether the category indicated by indication 902 matches a category (e.g., category 922) determined by one or more classifiers 920. Alternatively or additionally, verifying indication 902 or verifying the category indicated by indication 902 includes determining that the category determined by one or more classifiers 920 is a specific instance or subclass of the category indicated by indication 902. For example, indication 902 corresponding to the category "vehicle incident" can be verified by one or more classifiers 920 determining that category 922 corresponds to "car engine", "motorcycle engine", "brake sound", "car horn", "motorcycle horn", "train horn", "vehicle collision", etc. (which can be classified into different types of vehicle incidents).

[0149] Depending on some configurations, the accuracy of one or more classifiers 920 is improved by providing additional information related to the audio event, in addition to the audio data 904. For example, one or more processors 126 may be optionally configured to provide the audio data 904 and an audio category indication 902 as input to one or more classifiers 920 to determine the category 922 associated with the audio data 904. In embodiment 900, the audio data 904 includes one or more beamforming signals 910 (e.g., beamforming audio signals 148) input to one or more classifiers 920. In another example, one or more processors 126 may optionally be configured to receive direction data 912 corresponding to a sound source (e.g., direction of arrival information 142) from a first device, and provide audio data 904, direction data 912, and an indication of audio category 902 as input to one or more classifiers 920 to determine the category 922 associated with the audio data 904.

[0150] Optionally, one or more processors 126 are configured to generate one or more outputs in addition to the audio event information 145 to replace the audio event information 145, or to generate one or more outputs to be included in the audio event information 145, such as notification 930, control signal 932, classifier output 934, or a combination thereof. For example, in an implementation where the audio category (e.g., classification 922) corresponds to a vehicle event (e.g., a collision), one or more processors 126 may send a notification 930 of the vehicle event to one or more third devices based on the location of a first device (e.g., device 110) and the location of one or more third devices, as further described with reference to Figures 14 and 15. In another example, the user of device 120 may be participating in an outdoor event (e.g., hiking along a trail), and the audio category (e.g., classification 922) corresponds to a safety-related event (e.g., animal growling). In this example, one or more processors 126 can send a notification 930 of a safety-related event to one or more third devices (such as other hikers' phones or headsets) that are determined to be nearby based on location data associated with one or more third devices.

[0151] In another example, a control signal 932 is sent to the first device based on the classifier output 934. For illustration, the classifier output 934 may include a bit pattern, a numeric indicator, or a text label or description indicating the category 922 determined by one or more classifiers 920. In an illustrative example, the control signal 932 instructs the first device to perform an audio scaling operation. In another example, the control signal 932 instructs the first device to perform spatial processing based on the direction of the sound source. In yet another example, the control signal 932 instructs the first device to change its operating mode, such as switching from a media playback mode (e.g., playing streaming audio to the user of the first device) to a transparency mode (e.g., enabling the user of the first device to hear ambient sounds).

[0152] Optionally, one or more processors 126 are configured to perform one or more operations associated with the source of directional audio sound in a tracking audio scene, as further explained with reference to FIG16. In one example, one or more processors 126 may receive directional data 912 corresponding to a sound source detected by the first device. Based on audio events, one or more processors 126 may update a map of directional sound sources in the audio scene to generate an updated map. One or more processors 126 may send data corresponding to the updated map to one or more third devices geographically distant from the first device. As an illustrative and non-limiting example, one or more third devices may use the updated map to notify users of one or more third devices of sound sources detected near the first device, or to provide a shared audio experience for users participating in a shared virtual environment (e.g., in a virtual conference room).

[0153] FIG10 illustrates another embodiment 1000 of the device 120. Compared to embodiment 900 of FIG9, the audio event processing unit 154 (e.g., one or more classifiers 920) receives a multi-channel audio signal 1002 as input instead of a beamforming signal 910. For example, the multi-channel audio signal 1002 may include audio signals 170, 172 received in audio data 904, audio signals 190, 192 received from microphones 106, 108, or combinations thereof. The multi-channel audio signal 1002 may be provided as input to one or more classifiers 920 in combination with indication 902, direction data 912, or both.

[0154] For example, in some cases, beamforming data is unavailable, such as when an audio event is detected but the directionality of the audio event cannot be determined with sufficient accuracy (e.g., the sound is mainly diffuse or non-directional, or masked by other sounds interfering with beamforming). Examples of processing based on whether the signal transmitted between devices is an audio signal or a beamforming signal are illustrated with reference to Figures 12 and 13.

[0155] FIG11 illustrates an embodiment 1100 of the device 120 and FIG1150 showing audio processing that can be performed at the device 120. One or more processors 126 include a content separator 1120, which is configured to separate foreground signals from background signals in audio content based on embeddings corresponding to audio signals.

[0156] Content separator 1120 may include audio generation network 1122, which is configured to receive one or more embeddings 1104 corresponding to one or more signatures of a particular sound. For example, one or more embeddings 1104 may correspond to or include embedding 712 of FIG. 7. In some examples, one or more embeddings 1104 may include signatures of one or more audio events, signatures of a particular person's voice, etc. Audio generation network 1122 is also configured to receive audio data (shown as input mixed waveform 1102) that may include background and foreground sounds from various sound sources. Audio generation network 1122 is configured to determine whether the input mixed waveform 1102 includes any sound corresponding to one or more embeddings 1104, and to extract, isolate, or remove those specific sounds.

[0157] The target output 1106 is generated by the content separator 1120. The target output 1106 may include an audio signal corresponding to a specific sound. For example, a specific sound corresponding to one or more embeddings 1104 may be separated from the remaining sound in the input mixed waveform 1102 to produce the target output 1106. In one example, the specific sound may correspond to a foreground sound in the input mixed waveform 1102, and the target output 1106 may include a foreground sound with the background removed or attenuated.

[0158] In another example, the target output 1106 corresponds to a modified version of the input mixed waveform 1102 and may include the sound represented by the input mixed waveform 1102 and remaining after a particular sound is removed (or attenuated). For example, the particular sound may correspond to a foreground sound in the input mixed waveform 1102, and the target output 1106 may include the background sound remaining in the input mixed waveform 1102 after the foreground sound is removed (or attenuated).

[0159] In another example, the target output 1106 may include an audio signal containing a specific sound as a foreground sound, which has been removed from the background sound of the input mixed waveform 1102 and added to a different set of background sounds.

[0160] In Figure 1150, a first foreground sound (FG1) 1154, a second foreground sound (FG2) 1156, and a third foreground sound (FG3) 1158 are illustrated in an audio scene 1151 that includes a first atmosphere 1152 (e.g., background). A foreground extraction operation 1160 is performed by a content separator 1120 using a first embedding of one or more embeddings 1104 for the first foreground sound 1154, a second embedding of one or more embeddings 1104 for the second foreground sound 1156, and a third embedding of one or more embeddings 1104 for the third foreground sound 1158 to isolate the foreground sounds 1154, 1156, and 1158 from the first atmosphere 1152 (shown as isolated foreground sound 1162). Scene generation operation 1164 adds foreground sounds 1154, 1156, 1158 to audio scene 1171 with a second atmosphere 1172 (e.g., an updated audio scene). Scene generation operation 1164 may be performed by audio generation network 1122, content splitter 1120, one or more processors 1126, or a combination thereof.

[0161] In one example, the input mixed waveform 1102 represents audio data corresponding to audio scene 1151, which is processed by one or more processors 1126 to produce adjusted audio data (e.g., a target output 1106 including isolated foreground sound 1162), and the adjusted data is further adjusted by one or more processors 1126 (e.g., scene generation operation 1164) to produce an updated audio scene (e.g., audio scene 1171). Audio scene 1171 may include directional information associated with various objects and audio events (e.g., audio and events associated with other participants in a shared audio scene), as further described with reference to Figures 16-18.

[0162] The content separator 1120, which includes the audio generation network 1122, can isolate any target sound from the background and is not limited to separating speech from noise. In some embodiments, the content separator 1120 using the audio generation network 1122 achieves single-microphone target separation of specific audio events, speech, etc., and can overcome the limitations associated with conventional techniques that cannot distinguish audio sources.

[0163] Figure 12 illustrates a flowchart corresponding to method 1200, which can be executed by a first device such as device 110 (e.g., one or more processors 116) for sending information to a second device such as device 120.

[0164] Method 1200 includes processing one or more frames of audio signals in block 1202. For example, audio data 178 (e.g., frames of audio signals 170, 172) may be processed at arrival direction processing unit 132, audio event processing unit 134, acoustic environment processing unit 136, unit 138 or a combination thereof, as shown in FIG1.

[0165] Method 1200 includes determining at block 1204 whether processing of one or more frames of an audio signal triggers ambient detection. In some examples, ambient detection may include determining that an ambient change has been detected. In response to determining that ambient detection has occurred, method 1200 includes sending ambient information to a second device at block 1206. For example, device 110 sends ambient information 146 to device 120.

[0166] In response to a determination at block 1204 that no ambient detection has occurred, or after sending ambient information at block 1206, method 1200 includes determining at block 1208 whether processing of one or more frames of the audio signal causes an audio event to be detected. In response to a determination that an audio event has been detected, method 1200 includes sending audio event information to a second device at block 1210. For example, device 110 sends audio event information 144 to device 120.

[0167] Furthermore, in response to determining that an audio event has been detected, method 1200 includes determining at block 1212 whether valid direction-of-arrival information is available. For example, valid direction-of-arrival information may correspond to the detection of a sound source having a direction of arrival determined at a confidence level higher than a confidence threshold, to distinguish between discrete sound sources and diffuse sound without a distinguishable source. In a particular embodiment, valid direction-of-arrival information available for sound represented by one or more audio signals indicates that the sound comes from an identifiable direction (e.g., from a discrete sound source), and valid direction-of-arrival information unavailable for the sound indicates that the sound does not come from an identifiable direction. In response to determining that valid direction-of-arrival information is available at 1212, method 1200 includes sending direction-of-arrival information to a second device at block 1214. For example, device 110 sends direction-of-arrival information 142 to device 120.

[0168] In response to a decision at block 1208 that no audio event was detected, a decision at block 1212 that no valid direction of arrival information is available, or a decision at block 1214 that direction of arrival information is sent to the second device, method 1200 proceeds to a decision at block 1222 on whether to send one or more audio signals (e.g., audio signals 170, 172), one or more beamforming signals (e.g., beamforming audio signal 148), or not to send any audio signals to the second device.

[0169] Figure 12 illustrates several optional decision operations that can be used in some embodiments to determine at block 1220 whether to send one or more audio signals, one or more beamforming signals, or not to send audio signals to the second device.

[0170] At block 1230, it is determined whether at least one ambient detection or audio event detection has occurred. In response to the determination that no ambient detection has occurred and no audio event has been detected, method 1200 determines at block 1240 that no audio will be sent to the second device. Therefore, in this example, when there is no ambient detection and no audio event, the first device (e.g., device 110) does not transmit audio information to the second device (e.g., device 120) for additional processing.

[0171] Otherwise, in response to the determination that at least one of environmental detection or audio event detection has occurred, method 1200 includes determining at block 1232 whether the amount of power or bandwidth available for transmission to the second device is limited. For example, if the first device has available battery power below a power threshold, or if the available transmission bandwidth for transmitting audio data to the second device is below a transmission threshold, the first device may decide to conserve resources associated with transmitting audio data to the second device. Otherwise, the first device may operate in a preset (e.g., non-conserving) mode.

[0172] In response to the determination at block 1232 that power and transmission bandwidth are not limited, method 1200 includes sending an audio signal to a second device at block 1248. For example, device 110 may send audio signals 170, 172 to device 120.

[0173] Otherwise, in response to a determination at block 1232 that at least one of power or transmission bandwidth is limited, method 1200 includes determining at block 1234 whether the microphone at the second device is available for capturing audio data. For example, the microphone at the second device (e.g., microphones 106, 108) may be considered unusable if it is obstructed or blocked (e.g., in a user's pocket or bag), or if it is too far away to capture substantially the same audio information as the microphone at the first device.

[0174] In response to determining at block 1234 that the microphone at the second device is available, method 1200 includes determining at block 1236 whether a beamforming audio signal is available. For example, when ambient detection occurs based on diffused ambient sound rather than sound from a specific source whose direction can be located, beamforming is not performed at the first device. As another example, when an audio event is detected, but the direction of the sound source corresponding to the audio event cannot be determined with a confidence level greater than a threshold confidence level, a valid beamforming signal is not generated at the first device.

[0175] In response to the determination at block 1236 that no beamforming audio signal is available, method 1200 determines at block 1240 that no audio data should be sent to the second device. Otherwise, when it is determined at block 1236 that a beamforming audio signal is available, method 1200 proceeds to block 1242, where either a beamforming signal or no signal is sent to the second device. For example, because power or transmission resources are limited, but a microphone is available at the second device for audio capture and analysis, the first device may determine that no audio should be sent to the second device, and the second device may capture the audio for analysis at the second device. Otherwise, even though power or transmission resources are limited and a microphone is available for audio capture at the second device, the first device may decide to send a beamforming audio signal to the second device. In a particular implementation, the decision on whether to transmit a beamforming signal or not at block 1242 may be based at least in part on the amount of power or bandwidth available for the transmission of the beamforming signal (e.g., a comparison with one or more bandwidth thresholds or power thresholds may be performed to determine whether to transmit one or more beamforming audio signals).

[0176] Returning to block 1234, in response to the determination that the microphone of the second device is unavailable, method 1200 determines at block 1238 whether one or more beamforming audio signals are available. In response to the determination that one or more beamforming audio signals are available, method 1200 includes sending one or more beamforming audio signals at block 1244. Otherwise, in response to the determination that one or more beamforming audio signals are unavailable at block 1238, method 1200 includes sending a simplified signal to the second device at block 1246. For example, transmitting a simplified signal may include transmitting audio corresponding to a reduced number of microphone channels (e.g., transmitting a single audio signal 170 or 172), transmitting a reduced-resolution version of one or more microphone channels (e.g., a lower-resolution version of one or more audio signals 170, 172), or transmitting extracted audio feature data (e.g., feature data extracted from one or both of audio signals 170, 172, such as spectral information). This can provide useful information to a second device with reduced power and bandwidth usage compared to transmitting the full audio signals 170, 172.

[0177] Figure 13 illustrates a flowchart corresponding to method 1300, which can be executed by a second device such as device 120 (e.g., one or more processors 126) in response to receiving information from a first device such as device 110.

[0178] Method 1300 includes receiving data transmission from a first device at block 1302. Method 1300 includes determining at block 1304 whether the transmission includes audio signal data. As an example, the second device may parse the received data to determine whether one or more audio signals (e.g., audio signals 170, 172, one or more beamforming signals 148, or a combination thereof) have been received.

[0179] If the transmission does not include audio signal data, method 1300 may optionally include determining at block 1304 whether one or more microphones of the second device are available for audio capture. For example, if the microphones of the second device (e.g., microphones 106, 108) are obstructed or blocked (such as in a user's pocket or bag), or are located too far away to capture substantially the same audio information as the microphones at the first device, the microphones at the second device may be considered unusable.

[0180] In response to a determination at block 1304 that one or more microphones are unavailable, method 1300 may optionally include sending a microphone unavailable signal to the first device at 1306, and the method terminates at 1308. Otherwise, when one or more microphones are available, method 1300 may optionally include performing a data acquisition operation at the second device at block 1310 to acquire audio signals.

[0181] Method 1300 may optionally include determining at block 1312 whether the transmission includes environmental data. As an example, device 120 may parse the received data to determine whether environmental information 146 has been received. In response to a transmission including environmental data, method 1300 may optionally include performing environmental processing at 1314. For example, device 120 may process audio signals 170, 172, 190, 192 or combinations thereof at acoustic environmental processing unit 156 to generate environmental information 147.

[0182] Method 1300 includes determining at block 1320 whether the transmission includes audio event data. As an example, device 120 may parse the received data to determine whether audio event information 144 has been received. If the transmission does not include audio event data, processing of the data received in the transmission ends at 1322. In response to the transmission including audio event data, method 1300 may optionally include determining at block 1330 whether the transmission includes direction of arrival data. As an example, device 120 may parse the received data to determine whether direction of arrival information 142 has been received. In response to the transmission not including direction of arrival data, method 1300 may optionally include performing direction of arrival processing at 1332 to generate direction of arrival data. For example, device 120 may process audio signals 170, 172, 190, 192 or combinations thereof at direction of arrival processing unit 152 to generate direction of arrival information 143. However, if the transmission includes direction of arrival data, the direction of arrival processing at block 1332 is bypassed. Therefore, the second device can selectively bypass the arrival direction processing of the received audio data corresponding to the audio event based on whether it receives arrival direction information from the first device.

[0183] When arrival direction information is transmitted at block 1330, or after arrival direction information is generated at block 1332, method 1300 may optionally include determining at block 1340 whether the transmission includes beamforming data. As an example, device 120 may parse the received data to determine whether beamforming audio signal 148 is received. In response to the transmission not including beamforming data, method 1300 may optionally include performing a beamforming operation at 1342 to generate beamforming data. For example, device 120 may process audio signals 170, 172, 190, 192 or combinations thereof at beamforming unit 158 ​​to generate beamforming audio signal 149. However, if the transmission includes beamforming data, the execution of the beamforming operation at block 1342 is bypassed. Therefore, the second device may selectively bypass the beamforming operation based on whether the received audio data corresponds to a multi-channel microphone signal from the first device or a beamforming signal from the first device.

[0184] When beamforming data is transmitted at block 1340, or after beamforming data is generated at block 1342, method 1300 includes performing audio event processing at block 1350. For example, device 120 may process audio signals 170, 172, 190, 192 or combinations thereof at audio event processing unit 154 to generate audio event information 145.

[0185] By selectively bypassing one or more operations (such as direction of arrival processing or beamforming operations), method 1300 achieves reduced power consumption, reduced latency, or both associated with processing audio event data received from the first device.

[0186] Referring to FIG14, a particular illustrative example of a system configured to perform directional processing on multiple audio signals received from multiple microphones is disclosed, and is generally designated as 1400. System 1400 includes a vehicle 1410 coupled to a first microphone 1402 and a second microphone 1404. Although two microphones 1402, 1404 are illustrated, in other embodiments, additional microphones may be coupled to vehicle 1410. As a non-limiting example, eight (8) microphones may be coupled to vehicle 1410. In some embodiments, microphones 1402, 1404 are directional microphones. In other embodiments, one or both of microphones 1402, 1404 are omnidirectional microphones.

[0187] According to some embodiments, vehicle 1410 may be an autonomous vehicle. That is, vehicle 1410 can navigate without user interaction. According to other embodiments, vehicle 1410 may include one or more user-assisted modes (e.g., obstacle detection, obstacle avoidance, lane maintenance, speed control, etc.), and in some examples may be able to switch between user-assisted modes and autonomous modes. System 1400 also includes device 1420. According to one embodiment, device 1420 includes a second vehicle. According to another embodiment, device 1420 includes a server. As described below, vehicle 1410 may wirelessly communicate with device 1420 to perform one or more operations (such as autonomous navigation) based on sounds detected at vehicle 1410. In a particular embodiment, vehicle 1410 corresponds to device 110, and device 1420 corresponds to device 120.

[0188] The first microphone 1402 is configured to capture sound 1482 from one or more sources 1480. In the illustrative example of FIG. 14, source 1480 corresponds to another vehicle (such as a car). However, it should be understood that the vehicle is merely a non-limiting example of a sound source, and the techniques described herein can be implemented with other sound sources. When capturing sound 1482 from source 1480, the first microphone 1402 is configured to generate an audio signal 1470 representing the captured sound 1482. Similarly, the second microphone 1404 is configured to capture sound 1482 from one or more sources 1480. When capturing sound 1482 from source 1480, the second microphone 1404 is configured to generate an audio signal 1472 representing the captured sound 1482.

[0189] The first microphone 1402 and the second microphone 1404 may have different positions, different orientations, or both on the vehicle 1410. As a result, microphones 1402 and 1404 may capture sound 1482 at different times, with different reception phases, or both. For illustration, if the first microphone 1402 is closer to the source 1480 than the second microphone 1404, then the first microphone 1402 may capture sound 1482 before the second microphone 1404 captures it. As described below, if the positions and orientations of microphones 1402 and 1404 are known, the audio signals 1470 and 1472 generated by microphones 1402 and 1404, respectively, can be used to perform directional processing. That is, the vehicle 1410 can use audio signals 1470 and 1472 to determine the relative position of the source 1480, determine the direction of arrival of sound 1482, etc.

[0190] The vehicle 1410 includes a first input interface 1411, a second input interface 1412, memory 1414, and one or more processors 1416. The first input interface 1411 is coupled to one or more processors 1416 and configured to be coupled to a first microphone 1402. The first input interface 1411 is configured to receive an audio signal 1470 (e.g., a first microphone output) from the first microphone 1402 and can provide the audio signal 1470 to the processor 1416 as an audio frame 1474. The second input interface 1412 is coupled to one or more processors 1416 and configured to be coupled to a second microphone 1404. The second input interface 1412 is configured to receive an audio signal 1472 (e.g., a second microphone output) from the second microphone 1404 and can provide the audio signal 1472 to the processor 1416 as an audio frame 1476. Audio signals 1470, 1472, audio frames 1474, 1476, or both, may also be referred to herein as audio data 1478.

[0191] One or more processors 1416 include an arrival direction processing unit 1432, and optionally include an audio event processing unit 1434, a report generator 1436, a navigation instruction generator 1438, or a combination thereof. According to one embodiment, one or more elements of one or more processors 1416 may be implemented using dedicated circuitry. As a non-limiting example, one or more elements of one or more processors 1416 may be implemented using an FPGA, an ASIC, or the like. According to another embodiment, one or more elements of one or more processors 1416 may be implemented by executing instructions 1415 stored in memory 1414. For example, memory 1414 may be a non-transitory computer-readable medium storing instructions 1415 that can be executed by one or more processors 1416 to perform the operations described herein.

[0192] The direction of arrival processing unit 1432 can be configured to process a plurality of audio signals 1470, 1472 to generate direction of arrival information 1442 corresponding to the source 1480 of the sound 1482 represented by the audio signals 1470, 1472. In some embodiments, the direction of arrival processing unit 1432 is configured to operate in a manner similar to that of the direction of arrival processing unit 132 of FIG1. ​​In an illustrative and non-limiting example, the direction of arrival processing unit 1432 can select audio frames 1474, 1476 generated from each microphone 1402, 1404 that represent similar sounds (such as sound 1482 from source 1480). For example, the direction of arrival processing unit 1432 can process audio frames 1474, 1476 to compare sound characteristics and ensure that audio frames 1474, 1476 represent the same instance of sound 1482. In response to the determination that audio frames 1474 and 1476 represent the same instance of sound 1482, the direction of arrival processing unit 1432 can compare the timestamps of each audio frame 1474 and 1476 to determine which microphone 1402 and 1404 first captured the corresponding instance of sound 1482. If audio frame 1474 has an earlier timestamp than audio frame 1476, the direction of arrival processing unit 1432 can generate direction of arrival information 1442 indicating that source 1480 is closer to the first microphone 1402. If audio frame 1476 has an earlier timestamp than audio frame 1474, the direction of arrival processing unit 1432 can generate direction of arrival information 1442 indicating that source 1480 is closer to the second microphone 1404. Therefore, based on the similar timestamps of audio frames 1474 and 1476, the direction of arrival processing unit 1432 can locate sound 1482 and its corresponding source 1480. The timestamp from the audio frame of the additional microphone can be used to improve positioning in a similar manner to that described above.

[0193] In some embodiments, instead of the time difference described above, or in addition, one or more other techniques may be used to determine the direction of arrival information 1442, such as measuring the phase difference of the sound 1482 received at each microphone (e.g., microphones 1402 and 1404) of the microphone array of the vehicle 1410. In some embodiments, microphones 1402 and 1404 may operate as a microphone array or be included in a microphone array, and the direction of arrival information 1442 is generated based on the characteristics of the sound from each microphone in the microphone array (such as arrival time or phase) and based on the relative position and orientation of the microphones in the microphone array. In such embodiments, information about the sound characteristics or the captured audio data may be transmitted between the vehicle 1410 and the device 1420 for direction of arrival detection.

[0194] The audio event processing unit 1434 can be configured to process multiple audio signals 1470, 1472 to perform audio event detection in a manner similar to that of the audio event processing unit 134. For illustration, the audio event processing unit 1434 can process the sound characteristics of audio frames 1474, 1476 and compare these sound characteristics with a plurality of audio event models to determine whether an audio event has occurred. For example, the audio event processing unit 1434 can access a database (not shown) containing models of different audio events (such as car horns, train horns, pedestrian conversations, etc.). In response to a sound characteristic matching (or basic matching) of a specific model, the audio event processing unit 1434 can generate audio event information 1444 indicating that sound 1482 represents an audio event associated with a specific model. As a non-limiting example, the audio event could correspond to the sound of an approaching vehicle (e.g., source 1480).

[0195] The report generator 1436 can be configured to generate a report 1446 based on the direction of arrival information 1442 and the audio event information 1444. Therefore, the report 1446 can indicate at least one detected event and the direction of the detected event. In a scenario where microphones 1402 and 1404 capture multiple sounds from various directions, the report 1446 can indicate a list of events detected over a period of time and the direction information of the detected events.

[0196] Processor 1416 may be configured to send a report 1446 to device 1420. According to one embodiment, based on report 1446, device 1420 may send navigation instructions 1458 to vehicle 1410. Upon receiving navigation instructions 1458 from device 1420, processor 1416 may navigate (e.g., autonomously navigate) vehicle 1410 based on navigation instructions 1458. Alternatively or additionally, navigation instructions 1458 may be provided to the operator of vehicle 1410, such as visual or audible alerts or instructions for adjusting the operation of vehicle 1410. In some examples, navigation instructions 1458 indicate the path to be taken by vehicle 1410 (e.g., stopping on one side when it may be safe to allow an emergency vehicle to pass). In some examples, navigation instructions 1458 inform vehicle 1410 of the path of one or more other vehicles (e.g., a vehicle ahead has detected an accident and will slow down). The processor 1416 can autonomously navigate the vehicle 1410 to change its path (e.g., change its route or change its speed) to consider the path of one or more other vehicles.

[0197] According to another embodiment, based on or independently of report 1446, device 1420 may send a second report 1456 to vehicle 1410. In response to receiving the second report 1456, according to one embodiment, processor 1416 may navigate (e.g., autonomously navigate) vehicle 1410 based on both report 1446 and the second report 1456. According to another embodiment, in response to receiving the second report 1456, navigation instruction generator 1438 may be configured to generate navigation instructions 1448 to be used by processor 1416 to navigate vehicle 1410. In some examples, the second report 1456 indicates an event detected by another vehicle (e.g., a vehicle ahead detects a sound indicating an accident). Navigation instruction generator 1438 may generate navigation instructions 1448 to autonomously navigate vehicle 1410 to change its path, thereby avoiding the location of the event or changing its speed (e.g., slowing down). The processor 1416 can also send navigation instructions 1448 to the device 1420 to inform the device 1420 of the path of vehicle 1410. In some examples, navigation instructions 1448 indicate suggested paths (e.g., routes or speeds) for one or more other vehicles. For example, navigation instructions 1448 indicate that vehicle 1410 is slowing down and suggest that any vehicle within 20 feet of vehicle 1410 slow down or change its route.

[0198] Optionally, device 1420 may send a notification 1492 of an audio event (e.g., vehicle collision) to one or more other devices 1490 based on the location of vehicle 1410 and the location of one or more other devices 1490. In one example, notification 1492 corresponds to notification 930 of FIG. 9. As an illustrative and non-limiting example, one or more devices 1490 may include one or more other vehicles that can be determined to be near or approaching vehicle 1410, or combined with one or more other vehicles, to notify the vehicle of one or more detected audio events (e.g., siren, collision, etc.) near vehicle 1410.

[0199] The system 1400 of Figure 14 enables the vehicle 1410 to detect external sounds (such as sirens) and navigate accordingly. It should be understood that the use of multiple microphones can determine the location and relative distance of the sirens (e.g., source 1480) and can display the location and relative distance when the detected sirens are approaching or moving away.

[0200] FIG15 illustrates a specific illustrative configuration of a system 1500 including a vehicle 1510 (e.g., a first device) communicating with a device 1520 (e.g., a second device). The vehicle 1510 includes input interfaces 1412, 1411 of FIG14, memory 1414, and one or more processors 1416. In a particular embodiment, the vehicle 1510 corresponds to device 110, and device 1520 corresponds to device 120.

[0201] One or more processors 1416 include an implementation of an audio event processing unit 1434, wherein the generated audio event information 1444 indicates that a detected audio event corresponds to a vehicle event 1502 and an audio category 1504 associated with the vehicle event 1502. For example, the audio event processing unit 1434 may include one or more classifiers (such as one or more classifiers 610 of FIG. 6) configured to process audio data 1478 to determine an audio category 1504 corresponding to the sound 1482 represented by the audio data 1478 and associated with the vehicle event 1502.

[0202] One or more processors 1416 are configured to send audio data 1550 to device 1520 representing a sound associated with vehicle event 1502. For example, audio data 1550 may include audio data 1478, audio signals 1470, 1472, one or more beamforming audio signals pointing to a source 1480 of sound 1482, or a combination thereof. One or more processors 1416 are also configured to send an indication 1552 to device 1520 corresponding to an audio category 1504 associated with vehicle event 1502. For example, indication 1552 may correspond to indication 616 of FIG. 6 or FIG. 8.

[0203] Device 1520 includes memory 1514 configured to store instructions 1515, and also includes one or more processors 1516 coupled to memory 1514. The one or more processors 1516 are configured to receive audio data 1550 representing sound 1482 and an indication 1552 corresponding to an audio category 1504 associated with an event 1502 of the vehicle (e.g., a first device) from vehicle 1510. In certain embodiments, as a non-limiting example, device 1520 corresponds to another vehicle, server, or distributed computing (e.g., cloud-based) system.

[0204] One or more processors 1516 are also configured to process audio data 1550 at one or more classifiers 1530 to verify that the sound 1482 represented by the audio data 1550 corresponds to the transportation event 1502. For example, in a particular embodiment, one or more classifiers 1530 correspond to one or more classifiers 920 of FIG. 9. One or more processors 1516 are configured to send a notification 1492 of the transportation event 1502 to one or more devices 1490 based on the location of the transportation 1510 (e.g., a first device) and the location of one or more devices 1490 (e.g., one or more third devices).

[0205] FIG16 illustrates a specific implementation of device 120 (e.g., a second device), wherein one or more processors 126 are configured to update a map 1614 of directional sound sources based on audio events detected by a first device (e.g., device 110).

[0206] One or more processors 126 include an audio event processing unit 154, a map updater 1612, and an audio scene renderer 1618. The one or more processors 126 are configured to perform one or more operations associated with tracking directional audio sources in an audio scene. In one example, the one or more processors 126 may receive from a first device an indication 1602 of an audio category corresponding to an audio event (such as indication 616 in FIG. 6) and directional data 1604 corresponding to the sound source associated with the audio event (such as direction of arrival information 142).

[0207] One or more processors 126 can update the map 1614 of directional sound sources in the audio scene based on audio events to generate an updated map 1616. For example, when an audio event corresponds to a newly detected audio event, the map updater 1612 is configured to insert information corresponding to the audio event into the map 1614 to generate an updated map 1616. The inserted information may include, for example, an indication of the location of the sound source associated with the audio event, the type of the audio event (e.g., the audio category corresponding to the audio event), and information about the audio associated with the audio event (e.g., a link to audio signal data representing sound).

[0208] Optionally, one or more processors 126 may send data 1660 corresponding to the updated map 1616 to one or more third devices (shown as devices 1670, 1672, and 1674) geographically distant from the first device. Data 1660 enables devices 1670, 1672, and 1674 to update their local copies of the map 1614 so that users of devices 1670, 1672, or 1674 can be notified, accessed, or experienced sound associated with audio events.

[0209] In some embodiments, map 1614 (and the updated map 1616) corresponds to a database (such as a "crowdsourced" database) of sound events and locations distributed across a geographic area to notify vehicles or update vehicle navigation instructions to avoid specific sound events when a collision is detected nearby, as described in Figures 14 and 15. In other embodiments, map 1614 (and the updated map 1616) can be used for other applications, such as providing maps of sound events detected in the vicinity, towns, cities, etc. For example, maps of sound events associated with crime (e.g., shootings, shouts, sirens, broken glass, etc.) can be used by law enforcement agencies to plan resource allocation or to detect events worthy of investigation. As another example, maps of sound events can be associated with nature. For example, bird enthusiasts can use maps of various types of birds, which have been located based on the detection and classification of their specific calls.

[0210] In some embodiments, the audio scene renderer 1618 is configured to generate sound data corresponding to the three-dimensional sound scene based on the updated map 1616 for replay to the user of the first device. For example, the first device may correspond to an audio headset worn by the user (such as those shown with reference to FIG. 21), or a virtual reality, augmented reality, or mixed reality headset (such as those shown with reference to FIG. 25).

[0211] Figure 17 illustrates a graphic example of a 3D audio map 1700 of the audio scene surrounding a user 1702 wearing headphones. The 3D audio map 1700 may correspond to map 1614 of Figure 16 (or an updated map 1616). The 3D audio map 1700 includes a first vehicle 1710 moving generally toward the user 1702 and a second vehicle 1712 also moving generally toward the user. (The direction of movement of the moving audio sources is indicated by arrows.) Other sound sources include a dog barking 1714, a person speaking 1716, a pedestrian crossing timer 1718 counting down the remaining time to cross the street, and an artificial sound 1720 that has been edited into the 3D audio map 1700. For example, sound sources 1710-1718 may be real-world sound sources detected by the microphone of the headphones worn by user 1702, and artificial sound 1720 may be added to a specific location in the sound scene by an augmented reality engine (or game engine), such as a sound effect associated with a shop or restaurant at that location (e.g., a commercial advertising bell).

[0212] Figure 18 illustrates an example of a directional audio scene 1802 captured by sound event and environmental category detection (such as map 1614 based on Figure 16 (or updated map 1616)). The user 1804 is at the center of the directional audio scene 1802, and the figure illustrates multiple sets of virtual (or real) speakers associated with the sound field of the directional audio scene 1802, including a first representative speaker 1810 in a first set of speakers located substantially above and below the user 1804, a second representative speaker 1812 in a second set of speakers placed along the upper and lower perimeters of the directional audio scene 1802, and a third representative speaker 1814 in a third set of speakers located approximately at head height around the user 1804.

[0213] In a particular embodiment, operation 1820 (e.g., updating map 1614 to add or remove sound events based on type, direction, etc.) causes an updated directional audio scene 1830 that includes multiple virtual participants 1832, 1834 in addition to user 1804. For example, virtual participants 1832, 1834 may correspond to remote users sharing information about their respective local sound fields, which can be combined with directional audio scene 1802 to create an immersive shared virtual experience for user 1804 and various participants 1832, 1834. Such a shared virtual experience can be used in applications such as live travel channel guides or live meetings, parties, or event immersion for people who cannot attend in person due to social, health, or other constraints.

[0214] FIG19 illustrates an embodiment 1900 of at least one of devices 110, 120 as an integrated circuit 1902 including directional audio signal processing circuitry. For example, the integrated circuit 1902 includes one or more processors 1916. The one or more processors 1916 may correspond to one or more processors 116, one or more processors 126, one or more processors 202 of FIG2, the processing circuitry described with respect to FIGS. 3 to 5, one or more processors 1416, one or more processors 1516, or combinations thereof. The one or more processors 1916 include a directional audio signal processing unit 1990. The directional audio signal processing unit 1990 may include at least one element of processor 116, at least one element of processor 126, at least one element of processor 202, at least one element of headset 310, at least one element of headset 410, at least one element of mobile phone 420, at least one element of system 500, at least one element of processor 1416, at least one element of processor 1516, or combinations thereof.

[0215] The integrated circuit 1902 also includes an audio input 1904 (such as one or more bus interfaces) to enable audio data 178 to be received for processing. The integrated circuit 1902 also includes a signal output 1906 (such as a bus interface) to enable the transmission of directional audio signal data 1992. The directional audio signal data 1992 may correspond to at least one of the following: direction of arrival information 142, 143, audio event information 144, 145, environmental information 146, 147, beamforming audio signal 148, 149, direction information 250, first sound information 440, second sound information 442, context information 496, audio scaling angle 460, noise reduction parameter 462, direction of arrival information 542, audio event information 544, indication 616, indication 716, notification 930, control signal 932, classifier output 934, target output 1106, report 1446, 1456, navigation command 1448, 1458, notification 1492, indication 1552, audio data 1550, data 1660, or a combination thereof.

[0216] The integrated circuit 1902 serves as a component in a system including a microphone to perform directional audio signal processing, such as a mobile phone or tablet computer as shown in FIG. 20, an earphone as shown in FIG. 21, a wearable electronic device as shown in FIG. 22, a voice-controlled speaker system as shown in FIG. 23, a camera as shown in FIG. 24, a virtual reality headset, a mixed reality headset or an augmented reality headset as shown in FIG. 25, augmented reality glasses or mixed reality glasses as shown in FIG. 26, a set of in-ear devices as shown in FIG. 27, or a vehicle as shown in FIG. 28 or FIG. 29.

[0217] As an illustrative and non-limiting example, FIG20 illustrates an implementation 2000 in which device 120 is a mobile device 2002 (such as a telephone or tablet). The mobile device 2002 includes a third microphone 106 positioned primarily to capture the user's voice, one or more fourth microphones 108 positioned primarily to capture ambient sound, and a display screen 2004. A directional audio signal processing unit 1990 is integrated into the mobile device 2002 and is shown using dashed lines to indicate internal components of the mobile device 2002 that are generally not visible to the user. In a particular example, the directional audio signal processing unit 1990 may be used to generate directional audio signal data 1992, which is then processed to perform one or more operations at the mobile device 2002, such as launching a graphical user interface or otherwise (e.g., via an integrated "smart assistant" application) displaying additional information associated with the detected audio event at the display screen 2004.

[0218] Figure 21 illustrates an embodiment 2100 in which device 110 is an earphone device 2102. The earphone device 2102 includes a first microphone 102 positioned primarily to capture the user's voice and one or more second microphones 104 positioned primarily to capture ambient sound. A directional audio signal processing unit 1990 is integrated into the earphone device 2102. In a particular example, the directional audio signal processing unit 1990 may be used to generate directional audio signal data 1992 that causes the earphone device 2102 to perform one or more operations at the earphone device 2102, send the directional audio signal data 1992 to a second device (not shown) for further processing, or a combination thereof. The earphone device 2102 may be configured to (e.g., based on audio event information 144, audio event information 145, environmental information 146, environmental information 147, or a combination thereof) provide audible notifications of detected audio events or environmental conditions to the wearer of the earphone device 2102.

[0219] Figure 22 illustrates an embodiment 2200 in which at least one of devices 110, 120 is a wearable electronic device 2202 (shown as a "smartwatch"). A directional audio signal processing unit 1990, a first microphone 102, and one or more second microphones 104 are integrated into the wearable electronic device 2202. In a particular example, the directional audio signal processing unit 1990 may be used to generate directional audio signal data 1992, which is then processed to perform one or more operations at the wearable electronic device 2202, such as launching a graphical user interface or otherwise displaying other information associated with the detected audio event on the display screen 2204 of the wearable electronic device 2202. For illustration, the display screen 2204 of the wearable electronic device 2202 may be configured to display notifications based on the voice detected by the wearable electronic device 2202. In a specific example, wearable electronic device 2202 includes a tactile device that provides tactile notification (e.g., vibration) in response to the detection of an audio event. For example, the tactile notification may cause a user to look at wearable electronic device 2202 to view a notification of the detected audio event or environment displayed (such as based on audio event information 144, audio event information 145, environmental information 146, environmental information 147, or a combination thereof). Wearable electronic device 2202 can therefore alert users with hearing impairments or users wearing headphones to the detection of specific audio activity.

[0220] FIG23 is an embodiment 2300 in which at least one of devices 110, 120 is a wireless speaker and voice-activated device 2302. The wireless speaker and voice-activated device 2302 may have a wireless network connection and is configured to perform auxiliary operations. A directional audio signal processing unit 1990, a first microphone 102, one or more second microphones 104, a third microphone 106, a fourth microphone 108, or a combination thereof are included in the wireless speaker and voice-activated device 2302. The wireless speaker and voice-activated device 2302 also includes a speaker 2304. In a particular embodiment, the speaker 2304 corresponds to the speaker 336 of FIG3, the speaker 436 of FIG4, or both. During operation, the directional audio signal processing unit 1990 may be used to generate directional audio signal data 1992 and determine whether a keyword has been spoken. In response to the determination that a keyword has been spoken, the wireless speaker and voice-activated device 2302 may perform auxiliary operations (such as by performing an integrated auxiliary application). Assistive functions can include adjusting the temperature, playing music, turning on the lights, etc. For example, an assistive function may be performed in response to a command received after receiving a keyword or key phrase (e.g., "Hello assistant").

[0221] Figure 24 illustrates an embodiment 2400 in which at least one of devices 110, 120 is a portable electronic device corresponding to camera device 2402. A directional audio signal processing unit 1990, a first microphone 102, one or more second microphones 104, or a combination thereof, are included in camera device 2402. During operation, directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 and determine whether a keyword has been spoken. As an illustrative example, in response to the determination that a keyword has been spoken, camera device 2402 can perform operations in response to verbal user commands, such as adjusting image or video capture settings, image or video playback settings, or image or video capture instructions.

[0222] Figure 25 illustrates an embodiment 2500 in which device 110 includes a portable electronic device corresponding to an extended reality ("XR") headset 2502 (such as a virtual reality ("VR"), augmented reality ("AR"), or mixed reality ("MR") headset device. A directional audio signal processing unit 1990, a first microphone 102, one or more second microphones 104, or a combination thereof, are integrated into the headset 2502. In a particular embodiment, the headset 2502 includes a first microphone 102 positioned primarily to capture the user's voice and a second microphone 104 positioned primarily to capture ambient sound. The directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 based on audio signals received from the first microphone 102 and the second microphone 104 of the headset 2502. Visual peripheral devices are placed in front of the user's eyes to display augmented reality or virtual reality images or scenes to the user when wearing the headset 2502. In a specific example, the visual peripheral device is configured to display a notification indicating user voice detected in an audio signal. In another specific example, the visual peripheral device is configured to display a notification indicating a detected audio event, overlaid on displayed content (e.g., in virtual reality applications) or overlaid on the user's field of vision (e.g., in augmented reality applications), to visually indicate to the user the location of the sound source associated with the audio event. For illustration, the visual peripheral device may be configured to display notifications of detected audio events or the environment (e.g., based on audio event information 144, audio event information 145, environmental information 146, environmental information 147, or combinations thereof).

[0223] FIG26 illustrates an embodiment 2600 in which device 110 includes a portable electronic device corresponding to augmented reality or mixed reality glasses 2602. Glasses 2602 include a holographic projection unit 2604 configured to project visual data onto the surface of a lens 2606 or reflect visual data from the surface of the lens 2606 onto the wearer's retina. A directional audio signal processing unit 1990, a first microphone 102, one or more second microphones 104, or a combination thereof, are integrated into glasses 2602. The directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 based on audio signals received from the first microphone 102 and the second microphones 104. In a particular example, holographic projection unit 2604 is configured to display a notification indicating a user's voice detected in the audio signal. In a particular example, holographic projection unit 2604 is configured to display a notification indicating a detected audio event. For example, a notification can be overlaid on a specific location in the user's field of vision that corresponds to the location of the sound source associated with the audio event. For illustration, the sound can be perceived by the user as emanating from the direction of the notification. In an illustrative embodiment, the holographic projection unit 2604 is configured to display a notification of a detected audio event or environment (e.g., based on audio event information 144, audio event information 145, environmental information 146, environmental information 147, or a combination thereof).

[0224] Figure 27 illustrates an embodiment 2700 in which device 110 includes a portable electronic device corresponding to a pair of earplugs 2706 (including a first earplug 2702 and a second earplug 2704). Although earplugs are described, it should be understood that this technology can be applied to other in-ear or ear-hook playback devices.

[0225] The first earpiece 2702 includes a first microphone 2720, such as a high noise-to-noise ratio microphone positioned to capture the speech of the wearer of the first earpiece 2702, an array of one or more other microphones configured to detect ambient sound and spatially distributed to support beamforming (shown as microphones 2722A, 2722B and 2722C), an "interior" microphone 2724 located close to the wearer's ear canal (e.g., to aid in active noise cancellation), and a self-speech microphone 2726 (such as a bone conduction microphone configured to convert sound vibrations from the wearer's ear bones or skull into audio signals).

[0226] In a particular embodiment, the first microphone 2720 corresponds to microphone 102, and microphones 2722A, 2722B, and 2722C correspond to multiple instances of microphone 104, and audio signals generated by microphones 2720 and 2722A, 2722B, and 2722C are provided to the directional audio signal processing unit 1990. The directional audio signal processing unit 1990 can be used to generate directional audio signal data 1992 based on the audio signals. In some embodiments, the directional audio signal processing unit 1990 can also be configured to process audio signals from one or more other microphones (such as internal microphone 2724, self-voice microphone 2726, or both) of the first earpiece 2702.

[0227] The second earbud 2704 may be configured in a manner substantially similar to that of the first earbud 2702. In some embodiments, the directional audio signal processing unit 1990 of the first earbud 2702 is also configured to receive one or more audio signals generated by one or more microphones of the second earbud 2704 (e.g., via wireless transmission between earbuds 2702 and 2704, or via wired transmission in embodiments where earbuds 2702 and 2704 are coupled via transmission lines). In other embodiments, the second earbud 2704 further includes the directional audio signal processing unit 1990 to enable a user wearing a single earbud of earbuds 2702 and 2704 to perform the techniques described herein.

[0228] In some embodiments, earbuds 2702, 2704 are configured to automatically switch between various operating modes, such as a pass-through mode in which ambient sounds are played via speaker 2730, a replay mode in which non-ambient sounds (e.g., streaming audio corresponding to telephone conversations, media playback, video games, etc.) are played via speaker 2730, and an audio scaling or beamforming mode in which one or more ambient sounds are emphasized and / or other ambient sounds are suppressed for replay at speaker 2730. In other embodiments, earbuds 2702, 2704 may support fewer modes, or instead of the described modes, earbuds 2702, 2704 may also support one or more other modes in addition to the described modes.

[0229] In illustrative examples, earbuds 2702 and 2704 can automatically switch from playback mode to pass-through mode in response to the detection of the wearer's voice, and can automatically switch back to playback mode after the wearer stops speaking. In some examples, earbuds 2702 and 2704 can operate in two or more modes simultaneously (e.g., by performing audio scaling on specific ambient sounds (e.g., dog barking) and playing an audio-scaled sound superimposed on the playing sound while the wearer is listening to music (the volume of the music can be reduced while the audio-scaled sound is playing)). In this example, the wearer can be alerted to ambient sounds associated with an audio event without stopping the playback of music.

[0230] Figure 28 illustrates an implementation 2800 of the technology disclosed therein in a vehicle 2802 (shown as a manned or unmanned aerial vehicle (e.g., a package delivery drone)). A directional audio signal processing unit 2850 is integrated into the vehicle 2802. The directional audio signal processing unit 2850 includes or corresponds to the directional audio signal processing unit 1990 and may be further configured for autonomously navigating the vehicle 2802. The directional audio signal processing unit 2850 may include, for example, one or more processors 1416 of Figure 14, and the vehicle 2802 may correspond to the vehicle 1410. The directional audio signal processing unit 2850 may generate and execute navigation instructions, such as instructions for delivering information from an authorized user of the vehicle 2802, based on audio signals received from a first microphone 102 and a second microphone 104 of the vehicle 2802.

[0231] Figure 29 illustrates another embodiment 2900 in which vehicle 1410 or vehicle 1510 corresponds to vehicle 2902 (shown as an automobile). Vehicle 2902 includes a directional audio signal processing unit 2950. The directional audio signal processing unit 2950 includes or corresponds to directional audio signal processing unit 1990 and can be further configured for autonomously navigating vehicle 2902. Vehicle 2902 also includes a first microphone 102 and a second microphone 104. In some examples, one or more of the first microphone 102 and the second microphone 104 are located outside vehicle 2902 to capture ambient sounds, such as sirens and other vehicle sounds. In some embodiments, tasks such as detection of environmental information and audio events, autonomous navigation of vehicle 2902, etc., can be performed based on audio signals received from external microphones (e.g., first microphone 102 and second microphone 104).

[0232] In some examples, one or more of the first microphone 102 and the second microphone 104 are located inside the vehicle 2902 to capture sounds within the vehicle, such as voice commands or instructions for medical emergencies. In some implementations, tasks, such as autonomous navigation of the vehicle 2902, can be performed based on audio signals received from the internal microphones (e.g., the first microphone 102 and the second microphone 104). One or more operations of the vehicle 2902 can be initiated (such as by providing feedback or information via the display 2920 or one or more speakers (e.g., speaker 2910)) based on one or more detected keywords (e.g., "unlock", "start engine", "play music", "display weather forecast", or another voice command).

[0233] Referring to FIG30, a specific embodiment of a method 3000 for processing audio is illustrated. In this specific embodiment, one or more operations of the method 3000 are performed by a device 110, a system 200, an earphone 310, an earphone 410, a system 500, a vehicle 1410, a vehicle 1510, or a combination thereof.

[0234] Method 3000 includes receiving audio signals from a plurality of microphones at one or more processors of the first device at block 3002. For example, referring to FIG1, processor 130 may receive audio frames 174, 176 of audio signals 170, 172 from microphones 102, 104, respectively.

[0235] Method 3000 further includes processing audio signals in block 3004 to generate direction of arrival information corresponding to one or more sources of sound represented by one or more audio signals. For example, referring to FIG1, direction of arrival processing unit 132 may process audio frames 174, 176 to generate direction of arrival information 142 corresponding to source 180 of sound 182 represented by audio signals 170, 172.

[0236] Method 3000 further includes sending data at block 3006 to the second device based on direction-of-arrival information and a category or embedded data associated with the direction-of-arrival information. For example, modem 118 may send direction-of-arrival information 142 and one or both of indication 616 or indication 716 to device 120. The category may correspond to a category of a specific sound represented by an audio signal and associated with a specific audio event, and the embedding may include a signature or information corresponding to a specific sound or specific audio event, and may be configured to detect a specific sound or specific audio event in other audio signals by processing other audio signals. In some embodiments, method 3000 further includes sending a representation of an audio signal to the second device. For example, the representation of the audio signal may include one or more portions of audio signals 170, 172, one or more portions of beamforming audio signal 148, or a combination thereof. According to one embodiment of method 3000, sending data to device 120 may trigger the activation of one or more sensors 129.

[0237] In some embodiments, method 3000 includes processing signal data corresponding to an audio signal to determine a category or embedding. In one example, method 3000 includes performing beamforming operations on the audio signal (e.g., at beamforming unit 138) to generate signal data. In one example, the signal data is processed at one or more classifiers (such as one or more classifiers 610) to determine a category from multiple categories supported by one or more classifiers for sounds represented in the audio signal and associated with an audio event. The category (e.g., via indication 616) is sent to a second device (e.g., device 120).

[0238] In some embodiments, signal data is processed at one or more encoders (such as one or more encoders 710) to produce an embedding. The embedding corresponds to one or more sounds represented in an audio signal and associated with an audio event. The embedding is sent (such as via instruction 716) to a second device (e.g., device 120).

[0239] In some embodiments, method 3000 includes receiving data based on direction of arrival information and category at one or more processors of the second device. For example, modem 128 of device 120 may receive data and provide direction of arrival information 142 and indication 616 to one or more processors 126. Method 3000 may include obtaining audio data representing sound associated with the direction of arrival information and category at one or more processors of the second device. For example, one or more processors 126 may obtain one or more of audio signals 170, 172 from the first device, one or more of audio signals 190, 192 from a local microphone (e.g., microphones 106, 108), beamforming audio signal 148 or a combination thereof from the first device. Method 3000 may also include verifying the category at one or more processors of the second device based at least on the audio data and direction of arrival information, such as at audio event processing unit 154 or as described with reference to one or more classifiers 610.

[0240] In some embodiments, method 3000 includes receiving arrival direction information and embedded data at one or more processors of the second device. For example, the modem 128 of device 120 may receive the data and provide arrival direction information 142 and indication 716 to one or more processors 126. Method 3000 may also include processing audio data representing a sound scene at one or more processors of the second device based on the arrival direction information and embedded data to generate modified audio data corresponding to an updated sound scene. For example, one or more processors 126 may combine one or more embedded data 1104 and direction information 912 to process an input mixed waveform 1102 representing an audio scene 1151 to generate an updated audio scene 1171.

[0241] Method 3000 is capable of performing directional context-aware processing based on audio signals generated by multiple microphones. As a result, context detection for various use cases and determination of characteristics associated with the surrounding environment are achieved.

[0242] Referring to FIG31, a specific embodiment of the method 3100 for processing audio is illustrated. In this specific embodiment, one or more operations of the method 3100 are performed by the vehicle 1410 of FIG14.

[0243] Method 3100 includes receiving multiple audio signals from multiple microphones at one or more processors of the vehicle at block 3102. For example, referring to FIG14, processor 1416 may receive audio frames 1474 and 1476 of audio signals 1470 and 1472 from microphones 1402 and 1404, respectively.

[0244] Method 3100 further includes processing multiple audio signals in block 3104 to generate direction of arrival information corresponding to one or more sources of sound represented by one or more of the audio signals. For example, referring to FIG14, direction of arrival processing unit 1432 may process audio frames 1474, 1476 to generate direction of arrival information 1442 corresponding to source 1480 of sound 1482 represented by audio signals 1470, 1472.

[0245] Method 3100 further includes generating a report at block 3106 indicating at least one detected event and the direction of the detected event based on arrival direction information. For example, referring to FIG14, report generator 1436 may generate report 1446 indicating at least one detected event (from audio event information 1444) and the direction of the detected event (from arrival direction information 1442).

[0246] According to one embodiment, method 3100 may include sending a report to a second device (e.g., a second vehicle or server) and receiving navigation instructions or a second report from the second device. Based on the second report, the processor may generate navigation instructions to autonomously navigate the vehicle. If the second device sends navigation instructions, the processor may use the sent navigation instructions to autonomously navigate the vehicle.

[0247] Method 3100 enables vehicle 1410 to detect external sounds (such as sirens) and navigate accordingly. It should be understood that using multiple microphones can determine the location and relative distance of the siren sound (e.g., source 1480), and the location and relative distance can be displayed when the detected siren sound is approaching or moving away.

[0248] Referring to FIG32, a specific embodiment of a method 3200 for processing audio is illustrated. In a particular embodiment, one or more operations of method 3200 are performed by device 120 (such as at one or more processors 126).

[0249] Method 3200 includes receiving an indication of an audio category at one or more processors of the second device at block 3202, the indication being received from the first device and corresponding to an audio event. For example, one or more processors 126 of device 120 of FIG. 9 receive indication 902 (e.g., indication 616) from device 110 of FIG. 6.

[0250] Method 3200 includes processing audio data at one or more processors of the second device at block 3204 to verify that the sound represented by the audio data corresponds to the audio event. For example, one or more processors 126 of device 120 of FIG. 2 process audio data 904 to generate classification 922, thereby verifying that the sound represented by audio data 904 corresponds to the audio event. In one example, one or more processors 126 compare classification 922 with an audio category indicated by indication 902.

[0251] Optionally, method 3200 includes receiving audio data from a first device (e.g., device 110), and processing the audio data may optionally include providing the audio data as input to one or more classifiers to determine a classification associated with the audio data. For example, in some embodiments, the audio data 904 includes one or more portions of audio signals 170, 172, one or more portions of beamforming audio signals 148, or a combination thereof, and the audio data 904 is input to one or more classifiers 920. In some embodiments, the processing of the audio data further includes providing an indication of the audio category (e.g., indication 902) as a second input to one or more classifiers to determine a classification associated with the audio data.

[0252] Optionally, method 3200 includes sending control signals (such as control signal 932) to a first device (e.g., device 110) based on the outputs of one or more classifiers. In some embodiments, the control signals include audio scaling instructions. In some embodiments, the control signals include instructions for performing spatial processing based on the direction of the sound source.

[0253] In some embodiments, the audio category corresponds to a transportation event, and method 3200 may optionally include sending a notification of the transportation event to one or more third devices based on the location of the first device and the location of one or more third devices. For example, notification 1492 is sent to one or more devices 1490, as described with reference to Figures 14 and 15.

[0254] Optionally, method 3200 includes receiving directional data (such as directional data 912) corresponding to sound sources associated with an audio event from a first device (e.g., device 110). Method 3200 may include updating a map of directional sound sources in the audio scene based on the audio event to generate an updated map, such as that described in reference map updater 1612, and sending data corresponding to the updated map to one or more third devices geographically distant from the first device. For example, device 120 sends data 1660 to one or more of devices 1670, 1672, and 1674.

[0255] Optionally, method 3200 includes selectively bypassing arrival direction processing of received audio data corresponding to an audio event based on whether arrival direction information is received from a first device (e.g., device 110). For example, one or more processors 126 may selectively bypass performing the arrival direction processing shown in block 1332 of FIG. 13 based on determining at block 1330 of FIG. 13 that arrival direction information has been received in a transmission from the first device.

[0256] Optionally, method 3200 includes selectively bypassing beamforming operations based on whether the received audio data corresponds to a multi-channel microphone signal from a first device (e.g., device 110) or a beamforming signal from the first device. For example, one or more processors 126 may selectively bypass the beamforming operation shown at block 1342 of FIG. 13 based on a decision at block 1340 of FIG. 13 to transmit beamforming data (such as beamforming audio signal 148).

[0257] By receiving an indication of an audio category corresponding to an audio event and processing audio data to verify that the sound represented by the audio data corresponds to the audio event, method 3200 can perform distributed audio event detection, enabling a first stage (such as at a headset) to identify audio events with relatively high sensitivity and relatively low accuracy (e.g., due to power, memory, or computational constraints) compared to a second stage (such as at a mobile phone). The second stage can use higher power and more accurate audio event detection to verify the audio event and can convey detection results, control signals, etc., based on the detected audio event. As a result, accurate audio event detection can be provided to users of wearable electronic devices (such as headsets) without requiring the wearable electronic device to support the computational load, memory footprint, and power consumption associated with full-power audio event detection.

[0258] Referring to FIG33, a specific embodiment of the method 3300 for processing audio is illustrated. In one particular embodiment, one or more operations of the method 3300 are performed by device 120 (such as at one or more processors 126). In another particular embodiment, one or more operations of the method 3300 are performed by device 1520 (such as at one or more processors 1526).

[0259] Method 3300 includes receiving audio data from a first device and an indication from the first device at one or more processors of the second device at block 3302, the audio data corresponding to an audio category associated with a vehicle event. For example, device 1520 receives audio data 1550 and indication 1552 from vehicle 1510.

[0260] Method 3300 includes processing audio data at one or more classifiers in a second device (e.g., device 1520) at block 3304 to verify that the sound represented by the audio data corresponds to a vehicle event. For example, audio data 1550 is processed at one or more classifiers 1530 to determine a classification 1522.

[0261] Method 3300 includes sending a notification of a vehicle event to one or more third devices at block 3306 based on the location of a first device (e.g., vehicle 1510) and the locations of one or more third devices. For example, device 1520 sends notification 1592 to one or more devices 1490 based on the location of vehicle 1510 and the locations of one or more devices 1490.

[0262] Referring to FIG34, a specific embodiment of a method 3400 for processing audio is illustrated. In a particular embodiment, one or more operations of method 3400 are performed by device 110 (such as at one or more processors 116).

[0263] Method 3400 includes receiving one or more audio signals from one or more microphones at one or more processors of the first device at block 3402. For example, device 110 receives audio signals 170 and 172 from microphones 102 and 104, respectively.

[0264] Method 3400 includes processing one or more audio signals at one or more processors at block 3404 to determine whether the sound represented by one or more of the audio signals originates from an identifiable direction. For example, at block 1212 of FIG12, device 110 determines whether the processing of the audio signals at block 1202 of FIG12 has produced valid direction-of-arrival information about the source of the audio event.

[0265] Method 3400 includes selectively sending the direction of arrival information of the sound source to the second device based on the decision at block 3406. For example, device 110 selects whether to send the direction of arrival information to the second device based on whether valid direction of arrival information is available, as described in conjunction with blocks 1212 and 1214 of FIG12.

[0266] By selectively sending arrival direction information based on whether one or more of the sounds represented in the audio signal come from an identifiable direction, method 3400 can save power consumption and transmission resources that would otherwise be consumed by sending invalid or unreliable arrival direction information to a second device.

[0267] Referring to FIG35, a specific embodiment of a method 3500 for processing audio is illustrated. In a particular embodiment, one or more operations of method 3500 are performed by device 110 (such as at one or more processors 116).

[0268] Method 3500 includes receiving one or more audio signals from one or more microphones at one or more processors of the first device at block 3502. For example, device 110 receives audio signals 170 and 172 from microphones 102 and 104, respectively.

[0269] Method 3500 includes determining, at block 3504, at one or more processors, based on one or more criteria whether to send one or more audio signals to the second device or to send a beamforming audio signal generated based on one or more audio signals to the second device. For example, if beamforming audio signals are available at device 110, device 110 may determine whether to send one or more audio signals or whether to send beamforming audio signals based on criteria such as available power and bandwidth resources, as described with reference to block 1220 of FIG12. In an illustrative, non-limiting example where no microphone is available at the second device, if the available power or bandwidth for transmission to the second device exceeds a threshold, then as described in conjunction with block 1232 of FIG12, it is decided to transmit an audio signal (e.g., via the "No" path from block 1232); otherwise, it is decided to transmit a beamforming signal (e.g., via the "Yes" path from block 1232, the "No" path from block 1234, and the "Yes" path from block 1238).

[0270] Method 3500 includes sending audio data corresponding to one or more audio signals or beamforming audio signals to a second device at block 3506 based on the decision. Continuing the example above, device 110 may send an audio signal to device 120 at block 1248 of FIG. 12, or send a beamforming signal to device 120 at block 1244 of FIG. 12.

[0271] By selecting whether to transmit an audio signal or a beamforming signal based on one or more criteria such as power availability or transmission resources, method 3400 enables the transmitting device to appropriately determine whether to provide full audio resolution (e.g., by transmitting data corresponding to a complete set of microphone channels including the sound of interest) or to provide more refined targeted audio (e.g., by transmitting data corresponding to a single beamforming channel aimed at the sound source of interest).

[0272] Referring to FIG36, a specific embodiment of a method 3600 for processing audio is illustrated. In a specific embodiment, one or more operations of method 3600 are performed by device 120 (such as at one or more processors 126).

[0273] Method 3600 includes receiving, at block 3602, audio data representing sound, direction data corresponding to a sound source, and a classification of sound corresponding to an audio event at one or more processors of the second device, wherein the audio data, direction data, and classification are received from the first device. For example, one or more processors 126 of device 120 may receive the audio data 904 of FIG9 or FIG10, the indication 1602 of FIG16, and the direction data 1604 from device 110.

[0274] Method 3600 includes processing audio data at one or more processors at block 3604 to verify that the sound corresponds to an audio event. For example, audio event processing unit 154 processes audio data to verify the audio category indicated by indicator 1602.

[0275] Method 3600 includes updating a map of directional sound sources in an audio scene based on audio events at one or more processors at block 3606 to produce an updated map. For example, map updater 1612 updates map 1614 to produce an updated map 1616.

[0276] Method 3600 includes sending data corresponding to the updated map to one or more third devices geographically distant from the first device at block 3608. For example, updated map data 1660 is sent to devices 1670, 1672 and 1674 geographically distant from device 110.

[0277] By updating the map of directional sound sources in an audio scene and sending the updated map data to a geographically remote device, method 3600 enables applications such as virtual environments in which multiple participants are immersed in a shared sound scene, as described with reference to Figure 18.

[0278] The methods of Figures 12, 13, and 30-36 can be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processing unit (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, the methods of Figures 12, 13, and 30-36 can be executed by a processor that executes instructions, as described with reference to Figure 37.

[0279] Referring to FIG. 37, a block diagram illustrating a particular illustrative embodiment of the device is shown, and is generally designated as 3700. In various embodiments, device 3700 may have more or fewer components than shown in FIG. 37. In the illustrative embodiment, device 3700 may correspond to device 110, device 120, vehicle 1410, device 1420, vehicle 1510, or device 1520. In the illustrative embodiment, device 3700 may perform one or more operations described with reference to FIGS. 1-36.

[0280] In a particular embodiment, device 3700 includes processor 3706 (e.g., CPU). Device 3700 may include one or more additional processors 3710 (e.g., one or more DSPs). In a particular example, the processor(s) 116, 126 of FIG. 1 or the processor(s) 1416 of FIG. 1 correspond to processor 3706, processor 3710, or a combination thereof. Processor 3710 may include a voice and music transcoder-decoder (CODEC) 3708, which includes a voice transcoder ("vocoder") encoder 3736, a vocoder decoder 3738, a directional audio signal processing unit 1990, or a combination thereof.

[0281] Device 3700 may include memory 3786 and CODEC 3734. Memory 3786 may include instructions 3756, which may be executed by one or more additional processors 3710 (or processor 3706) to implement the functions described with reference to the directional audio signal processing unit 1990. In a particular configuration, memory 3786 corresponds to memory 114, memory 124 of FIG. 1, memory 1414 of FIG. 14, or a combination thereof. In a particular configuration, instructions 3756 include instructions 115 and 125 of FIG. 1, instructions 1415 of FIG. 14, or a combination thereof. Device 3700 may include a modem 3770 coupled to antenna 3752 via transceiver 3750. Modem 3770 may be configured to transmit signals to a second device (not shown). According to a particular embodiment, modem 3770 may correspond to modem 128 of FIG. 1.

[0282] Device 3700 may include a display 3728 coupled to display controller 3726. Speaker 3792, first microphone 102, and second microphone 104 may be coupled to CODEC 3734. CODEC 3734 may include digital-to-analog converter (DAC) 3702, analog-to-digital converter (ADC) 3704, or both. In a particular embodiment, CODEC 3734 may receive analog signals from first microphone 102 and second microphone 104, convert the analog signals into digital signals using analog-to-digital converter 3704, and provide the digital signals to voice and music transcoder 3708. Voice and music transcoder 3708 may process digital signals, and the digital signals may be further processed by directional audio signal processing unit 1990. In a particular embodiment, voice and music transcoder 3708 may provide digital signals to CODEC 3734. The CODEC 3734 can use a digital-to-analog converter 3702 to convert digital signals into analog signals and can provide the analog signals to a speaker 3792.

[0283] In a particular embodiment, device 3700 may be included in a system-in-package or system-on-a-chip (SoC) 3722. In a particular embodiment, memory 3786, processor 3706, processor 3710, display controller 3726, CODEC 3734, and modem 3770 are included in a system-in-package or SoC 3722. In a particular embodiment, input device 3730 and power supply 3744 are coupled to SoC 3722. Furthermore, in a particular embodiment, as shown in FIG37, display 3728, input device 3730, speaker 3792, first microphone 102, second microphone 104, antenna 3752, and power supply 3744 are external to SoC 3722. In a particular embodiment, each of the display 3728, input device 3730, speaker 3792, first microphone 102, second microphone 104, antenna 3752, and power supply 3744 may be coupled to elements of the system-on-chip device 3722, such as an interface (e.g., input interface 121 or input interface 122) or a controller.

[0284] Device 3700 may include smart speakers, speaker strips, mobile communication devices, smartphones, cellular phones, laptops, computers, tablets, personal digital assistants, display devices, televisions, game consoles, music players, radios, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, vehicles, headphones, augmented reality headphones, mixed reality headphones, virtual reality headphones, aircraft, home automation systems, voice-activated devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, vehicles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof.

[0285] In conjunction with the described embodiments, an apparatus includes a component for receiving audio signals from a plurality of microphones. For example, the component for receiving audio signals may correspond to input interface 112, input interface 111, processor 116 or elements thereof, input interface 121, input interface 122, processor 126 or elements thereof, first processing domain 210 or elements thereof, second processing domain 220 or elements thereof, earphone 310 or elements thereof, earphone 410 or elements thereof, spatial filtering processing unit 502, audio input 1904, one or more processors 1916, directional audio signal processing unit 1990, one or more processors 3710, one or more other circuits or elements configured to receive audio signals from a plurality of microphones, or any combination thereof.

[0286] The device also includes components for processing audio signals to generate arrival direction information corresponding to one or more sources of sound represented in one or more audio signals. For example, the components for processing may correspond to processor(s) 116 or components thereof, processor(s) 126 or elements thereof, first processing domain 210 or elements thereof, second processing domain 220 or elements thereof, earphone 310 or elements thereof, earphone 410 or elements thereof, spatial filtering processing unit 502, audio event processing unit 504, directional audio signal processing unit 1990, one or more processors 1916, one or more processors 3710, one or more other circuits or elements configured to process audio signals, or any combination thereof.

[0287] The apparatus also includes components for transmitting to the second device data based on direction-of-arrival information and category or embedded data associated with the direction-of-arrival information. For example, the components for transmitting may correspond to modem 118, modem 128, signal output 1906, directional audio signal processing unit 1990, one or more processors 1916, modem 3770, transceiver 3750, antenna 3752, one or more other circuits or elements configured to transmit data and category or embedded data, or any combination thereof.

[0288] In conjunction with the described embodiments, an apparatus includes a component for receiving multiple audio signals from multiple microphones. For example, the component for receiving multiple audio signals may correspond to an input interface 1412, an input interface 1411, one or more processors 1416 or elements thereof, a directional audio signal processing unit 2850, a directional audio signal processing unit 2950, ​​one or more processors 3710, one or more other circuits or elements configured to receive multiple audio signals from multiple microphones, or any combination thereof.

[0289] The device also includes components for processing multiple audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals. For example, the components for processing include one or more processors 1416 or elements thereof, directional audio signal processing unit 2850, directional audio signal processing unit 2950, ​​one or more processors 3710, one or more other circuits or elements configured to process multiple audio signals, or any combination thereof.

[0290] The device also includes components for generating a report indicating at least one detected event and the direction of the detected event based on arrival direction information. For example, the components for generating the report include one or more processors 1416 or elements thereof, directional audio signal processing unit 2850, directional audio signal processing unit 2950, ​​one or more processors 3710, one or more other circuits or elements configured to generate the report, or any combination thereof.

[0291] In conjunction with the described embodiments, an apparatus includes a component for receiving an indication of an audio category received from a remote device and corresponding to an audio event. For example, the component for receiving the indication may correspond to a modem 128, one or more processors 126, one or more processors 1516, an audio input 1904, one or more processors 1916, an antenna 3752, a transceiver 3750, a modem 3770, a processor 3706, one or more processors 3710, one or more other circuits or elements configured to receive the indication, or any combination thereof.

[0292] The device also includes components for processing audio data to verify that the sound represented by the audio data corresponds to the audio event. For example, the components for processing the audio data may correspond to one or more processors 126, one or more processors 1516, one or more processors 1916, processor 3706, one or more processors 3710, one or more other circuits or elements configured to process audio data to verify that the sound represented by the audio data corresponds to the audio event, or any combination thereof.

[0293] In some embodiments, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 114 or memory 3786) includes instructions (e.g., instruction 115 or instruction 3756) that, when executed by one or more processors (e.g., one or more processors 116, one or more processors 3710, or processor 3706), cause one or more processors to receive audio signals (e.g., audio signals 170, 172) from a plurality of microphones (e.g., microphones 102, 104). When executed by one or more processors, the instructions also cause one or more processors to process the audio signals to generate arrival direction information (e.g., arrival direction information 142) corresponding to one or more sources (e.g., one or more sources 180) of one or more sounds (e.g., sound 182) represented in the audio signals. When executed by one or more processors, these instructions also cause one or more processors to send to a second device (e.g., device 120) data based on arrival direction information and categories or embedded data associated with the arrival direction information.

[0294] In some embodiments, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 3786) includes instructions (e.g., instructions 3756) that, when executed by one or more processors (e.g., one or more processors 3710 or processor 3706) of a vehicle (e.g., vehicle 1410), cause the one or more processors to receive multiple audio signals (e.g., audio signals 1470, 1472) from multiple microphones (e.g., microphones 1402, 1404). When executed by one or more processors, the instructions also cause the one or more processors to process the multiple audio signals to generate arrival direction information (e.g., arrival direction information 1442) corresponding to one or more sources (e.g., one or more sources 1480) of one or more sounds (e.g., sound 1482) represented in the audio signals. When executed by one or more processors, these instructions also cause one or more processors to generate a report (e.g., report 1446) based on arrival direction information, indicating at least one detected event and the direction of the detected event.

[0295] In some embodiments, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 124, memory 1514, or memory 3786) includes instructions (e.g., instruction 125, instruction 1515, or instruction 3756) that, when executed by one or more processors (e.g., one or more processors 126, one or more processors 1516, one or more processors 3710, or processor 3706), cause one or more processors to receive from the first device an indication of an audio category corresponding to an audio event (e.g., indication 902, indication 1552, or indication 1602).

[0296] This case includes the following first set of examples.

[0297] Example 1 includes a first device comprising: a memory configured to store instructions; and one or more processors configured to: receive a plurality of audio signals from a plurality of microphones; process the plurality of audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and send data based on the direction-of-arrival information to a second device.

[0298] Example 2 includes a first device according to Example 1, wherein memory and one or more processors are integrated into a headset device, and wherein the second device corresponds to a mobile phone.

[0299] Example 3 includes a first device according to Example 1, wherein memory and one or more processors are integrated into a mobile phone, and wherein a second device corresponds to a headset device.

[0300] Example 4 includes a first device according to any one of Examples 1 to 3, wherein data sent to a second device triggers the activation of one or more sensors at the second device.

[0301] Example 5 includes a first device according to any one of Examples 1 to 4, wherein at least one of the one or more sensors includes a non-audio sensor.

[0302] Example 6 includes a first device according to any one of Examples 1 to 5, wherein the non-audio sensor includes a 360-degree camera.

[0303] Example 7 includes a first device according to any one of Examples 1 to 6, wherein the non-audio sensor includes a laser radar sensor.

[0304] Example 8 includes a first device according to any one of Examples 1 to 7, wherein one or more processors include a first processing domain that operates in a low-power state.

[0305] Example 9 includes a first device according to any one of Examples 1 to 8, wherein one or more processors further include a second processing domain operating in a high-power state, the second power domain being configured to process a plurality of audio signals to generate arrival direction information.

[0306] Example 10 includes a first device according to any one of Examples 1 to 9, wherein one or more processors are further configured to: process a plurality of audio signals to perform audio event detection; and send data corresponding to the detected audio event to a second device.

[0307] Example 11 includes a first device according to any one of Examples 1 to 9, wherein one or more processors are further configured to: generate event data corresponding to a detected audio event based on an audio event detection operation; and send the event data to a second device.

[0308] Example 12 includes a first device according to any one of Examples 1 to 11, wherein one or more processors are further configured to: process a plurality of audio signals to perform acoustic environment detection; and send data corresponding to the detected environment to a second device.

[0309] Example 13 includes a first device according to any one of Examples 1 to 11, wherein one or more processors are further configured to generate environmental data corresponding to the detected environment based on acoustic environment detection operations.

[0310] Example 14 includes a first device according to any one of Examples 1 to 13, wherein one or more processors are further configured to: perform spatial processing on a plurality of audio signals based on direction of arrival information to generate a beamforming audio signal; and transmit the beamforming audio signal to a second device.

[0311] Example 15 includes a first device according to any one of Examples 1 to 14, wherein one or more processors are further configured to adjust the focus of at least one of a plurality of microphones based on direction of arrival information.

[0312] Example 16 includes a first device according to any one of Examples 1 to 15, and further includes a modem, wherein data is transmitted to a second device via the modem.

[0313] Example 17 includes a first device according to any one of Examples 1 to 16, wherein one or more processors are further configured to transmit a representation of a plurality of audio signals to a second device.

[0314] Example 18 includes the first device according to Example 17, wherein the representation of a plurality of audio signals corresponds to one or more beamforming audio signals.

[0315] Example 19 includes a first device according to any one of Examples 1 to 18, wherein one or more processors are further configured to generate a user interface output indicating at least one of an environmental event or an acoustic event.

[0316] Example 20 includes a first device according to any one of Examples 1 to 19, wherein one or more processors are further configured to receive data indicating acoustic events from a second device.

[0317] Example 21 includes a first device according to any one of Examples 1 to 20, wherein one or more processors are further configured to receive data indicating environmental events from a second device.

[0318] Example 22 includes a first device according to any one of Examples 1 to 21, wherein one or more processors are further configured to receive data indicating beamforming audio signals from a second device.

[0319] Example 23 includes a first device according to any one of Examples 1 to 22, wherein one or more processors are further configured to: receive directional information associated with a plurality of audio signals from a second device; and perform an audio scaling operation based on the directional information.

[0320] Example 24 includes a first device according to any one of Examples 1 to 23, wherein one or more processors are further configured to: receive directional information associated with a plurality of audio signals from a second device; and perform noise reduction operation based on the directional information.

[0321] Example 25 includes a first device according to any one of Examples 1 to 24, and also includes a plurality of microphones.

[0322] Example 26 includes a first device according to any one of Examples 1 to 25, and further includes at least one speaker configured to output sound associated with at least one of a plurality of audio signals.

[0323] Example 27 includes a first device according to any one of Examples 1 to 26, wherein one or more processors are integrated in a vehicle.

[0324] Example 28 includes a first device according to any one of Examples 1 to 27, wherein the data based on arrival direction information includes a report indicating at least one detected event and the direction of the detected event.

[0325] Example 29 includes a method for processing audio, the method comprising: receiving, at one or more processors of a first device, a plurality of audio signals from a plurality of microphones; processing the plurality of audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and sending data based on the direction-of-arrival information to a second device.

[0326] Example 30 includes the method according to Example 29, and further includes: processing a plurality of audio signals to perform audio event detection; and sending data corresponding to the detected audio event to a second device.

[0327] Example 31 includes the method according to Example 30, wherein audio event detection includes processing one or more of a plurality of audio signals at one or more classifiers to determine a category for a sound represented by one or more of the audio signals from a plurality of categories supported by one or more classifiers, wherein data corresponding to the detected audio event includes an indication of the category.

[0328] Example 32 includes the method according to any one of Examples 29 to 31, further comprising: processing a plurality of audio signals to perform acoustic environment detection; and sending data corresponding to the detected environment to a second device.

[0329] Example 33 includes the method according to any one of Examples 29 to 32, wherein data is transmitted to a second device via a modem.

[0330] Example 34 includes the method according to any one of Examples 29 to 33, and further includes a representation of sending a plurality of audio signals to a second device.

[0331] Example 35 includes the method according to any one of Examples 29 to 34, wherein data based on arrival direction information sent to the second device triggers the activation of one or more sensors at the second device.

[0332] Example 36 includes the method according to any one of Examples 29 to 35, wherein at least one of the one or more sensors includes a non-audio sensor.

[0333] Example 37 includes an apparatus comprising: a memory configured to store instructions; and a processor configured to execute instructions to perform the method according to any one of claims 29 to 36.

[0334] Example 38 includes an apparatus comprising a non-transitory computer-readable medium containing instructions which, when executed by one or more processors of the first apparatus, cause the one or more processors to perform the method according to any one of claims 29 to 36.

[0335] Example 39 includes an apparatus comprising components for performing the method according to any one of claims 29 to 36.

[0336] Example 40 includes a non-transitory computer-readable medium containing instructions that, when executed by one or more processors of a first device, cause the one or more processors to: receive a plurality of audio signals from a plurality of microphones; process the plurality of audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and send data based on the direction-of-arrival information to a second device.

[0337] Example 41 includes a non-transitory computer-readable medium according to Example 40, wherein data sent to a second device triggers the activation of one or more sensors at the second device.

[0338] Example 42 includes a non-transitory computer-readable medium according to Example 41 or 42, wherein at least one of the one or more sensors includes a non-audio sensor.

[0339] Example 43 includes a non-transitory computer-readable medium according to any one of Examples 40 to 42, wherein the instructions are executable to further cause one or more processors to send a plurality of audio signals to a second device.

[0340] Example 44 includes a non-transitory computer-readable medium according to Example 43, wherein the representation of a plurality of audio signals corresponds to one or more beamforming audio signals.

[0341] Example 45 includes a first device comprising: means for receiving a plurality of audio signals from a plurality of microphones; means for processing the plurality of audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and means for transmitting data based on the arrival direction information to a second device.

[0342] Example 46 includes a vehicle comprising: a memory configured to store instructions; and one or more processors configured to execute instructions to: receive a plurality of audio signals from a plurality of microphones; process the plurality of audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and generate a report based on the arrival direction information indicating at least one detected event and the direction of the detected event.

[0343] Example 47 includes a vehicle according to Example 46, wherein one or more processors are further configured to send a report to a second device.

[0344] Example 48 includes a means of transport according to any one of Examples 46 to 47, wherein the second device includes a second means of transport.

[0345] Example 49 includes a vehicle according to any one of Examples 46 to 48, wherein the second device includes a server.

[0346] Example 50 includes a vehicle according to any one of Examples 46 to 49, wherein one or more processors are further configured to: receive navigation instructions from a second device; and perform navigation based on the navigation instructions.

[0347] Example 51 includes a vehicle as described in any one of tool examples 46 to 50, wherein one or more processors are further configured to: receive a second report from a second device; and perform navigation based on the report and the second report.

[0348] Example 52 includes a vehicle according to any one of Examples 46 to 51, wherein one or more processors are further configured to: receive a second report from a second device; generate navigation instructions based on the second report; and send the navigation instructions to the second device.

[0349] Example 53 includes a vehicle according to any one of Examples 46 to 52, wherein a list of events detected over a period of time and directional information of the detected events is reported.

[0350] Example 54 includes a method for processing audio, the method comprising: receiving a plurality of audio signals from a plurality of microphones at one or more processors of a vehicle; processing the plurality of audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and generating a report based on the arrival direction information indicating at least one detected event and the direction of the detected event.

[0351] Example 55 includes the method according to Example 54, and further includes sending a report to a second device.

[0352] Example 56 includes the method according to any one of Examples 54 to 55, wherein the second device includes a second vehicle.

[0353] Example 57 includes the method according to any one of Examples 54 to 56, wherein the second device includes a server.

[0354] Example 58 includes the method according to any one of Examples 54 to 57, further comprising: receiving navigation instructions from a second device; and performing navigation based on the navigation instructions.

[0355] Example 59 includes the method according to any one of Examples 54 to 58, further comprising: receiving a second report from a second device; and performing navigation based on the report and the second report.

[0356] Example 60 includes the method according to any one of Examples 54 to 59, further comprising: receiving a second report from a second device; generating navigation instructions based on the second report; and sending the navigation instructions to the second device.

[0357] Example 61 includes the method according to any one of Examples 54 to 60, wherein a list indicating events detected over a period of time and directional information of the detected events is reported.

[0358] Example 62 includes a non-transitory computer-readable medium containing instructions that, when executed by one or more processors of a vehicle, cause one or more processors to: receive multiple audio signals from multiple microphones; process the multiple audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and generate a report based on the arrival direction information indicating at least one detected event and the direction of the detected event.

[0359] Example 63 includes a non-transitory computer-readable medium according to Example 62, wherein when executed by one or more processors, the instructions also cause one or more processors to send a report to a second device.

[0360] Example 64 includes a non-transitory computer-readable medium according to any one of Examples 62 to 63, wherein the second device includes a second vehicle.

[0361] Example 65 includes a non-transitory computer-readable medium according to any one of Examples 62 to 64, wherein the second device includes a server.

[0362] Example 66 includes a non-transitory computer-readable medium according to any one of Examples 62 to 65, wherein when executed by one or more processors, the instructions also cause one or more processors to: receive navigation instructions from a second device; and perform navigation based on the navigation instructions.

[0363] Example 67 includes a non-transitory computer-readable medium according to any one of Examples 62 to 66, wherein when executed by one or more processors, the instructions also cause one or more processors to: receive a second report from a second device; and perform navigation based on the report and the second report.

[0364] Example 68 includes a non-transitory computer-readable medium according to any one of Examples 62 to 67, wherein when executed by one or more processors, the instructions further cause one or more processors to: receive a second report from a second device; generate navigation instructions based on the second report; and send the navigation instructions to the second device.

[0365] Example 69 includes a non-transitory computer-readable medium according to any one of Examples 62 to 68, wherein a list of events detected over a period of time and directional information of the detected events is reported.

[0366] Example 70 includes a means of transport comprising: means for receiving a plurality of audio signals from a plurality of microphones; means for processing the plurality of audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and means for generating a report based on the arrival direction information indicating at least one detected event and the direction of the detected event.

[0367] Example 71 includes the vehicle according to Example 70, and also includes a component for sending a report to a second device.

[0368] Example 72 includes a means of transport according to any one of Examples 70 to 71, wherein the second device includes a second means of transport.

[0369] Example 73 includes a vehicle according to any one of Examples 70 to 72, wherein the second device includes a server.

[0370] Example 74 includes a vehicle according to any one of Examples 70 to 73, wherein a list of events detected over a period of time and directional information of the detected events is reported.

[0371] Example 75 includes a vehicle according to any one of Examples 70 to 74, and further includes a device that performs autonomous navigation based on a report.

[0372] This case includes the following second set of examples.

[0373] According to Example 1, a first device includes: a memory configured to store instructions; and one or more processors configured to: receive audio signals from a plurality of microphones; process the audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and send to a second device data based on the arrival direction information and category or embedded data associated with the arrival direction information.

[0374] Example 2 includes the first device according to Example 1, wherein one or more processors are further configured to process signal data corresponding to the audio signal to determine the category or embedding.

[0375] Example 3 includes the first device according to Example 2, wherein one or more processors are further configured to perform beamforming operations on an audio signal to generate signal data.

[0376] Example 4 includes a first device according to Example 2 or Example 3, wherein one or more processors are further configured to process signal data at one or more classifiers to determine a category from a plurality of categories supported by one or more classifiers for sounds represented in one or more audio signals and associated with audio events, and wherein the category is sent to a second device.

[0377] Example 5 includes a first device according to any one of Examples 2 to 4, wherein one or more processors are further configured to process signal data at one or more encoders to produce an embedding corresponding to one or more sounds represented in an audio signal and associated with an audio event, and wherein the embedding is sent to a second device.

[0378] Example 6 includes a first device according to any one of Examples 1 to 5, wherein one or more processors are further configured to process image data at one or more encoders to produce an embedding corresponding to an object represented by image data and associated with an audio event, and wherein the embedding is sent to a second device.

[0379] Example 7 includes the first device according to Example 6, and also includes one or more cameras configured to generate image data.

[0380] Example 8 includes a first device according to any one of Examples 1 to 7, wherein one or more processors are further configured to generate environmental data corresponding to the detected environment based on acoustic environment detection operations.

[0381] Example 9 includes a first device according to any one of Examples 1 to 8, wherein one or more processors are further configured to: perform spatial processing on an audio signal based on direction-of-arrival information to generate one or more beamforming audio signals; and transmit one or more beamforming audio signals to a second device.

[0382] Example 10 includes a first device according to any one of Examples 1 to 9, wherein memory and one or more processors are integrated into a headset device, and wherein the second device corresponds to a mobile phone.

[0383] Example 11 includes a first device according to any one of Examples 1 to 9, wherein one or more processors are integrated in a vehicle.

[0384] Example 12 includes a first device according to any one of Examples 1 to 11, and further includes a modem, wherein data is transmitted to a second device via the modem.

[0385] Example 13 includes a first device according to any one of Examples 1 to 12, wherein one or more processors are further configured to transmit an audio signal representation to a second device.

[0386] Example 14 includes a first device according to Example 13, wherein the representation of an audio signal corresponds to one or more beamforming audio signals.

[0387] Example 15 includes a first device according to any one of Examples 1 to 14, wherein one or more processors are further configured to generate a user interface output indicating at least one of an environmental event or an acoustic event.

[0388] Example 16 includes a first device according to any one of Examples 1 to 15, wherein one or more processors are further configured to receive data indicating acoustic events from a second device.

[0389] Example 17 includes a first device according to any one of Examples 1 to 16, wherein one or more processors are further configured to: receive directional information associated with an audio signal from a second device; and perform an audio scaling operation based on the directional information.

[0390] Example 18 includes a first device according to any one of Examples 1 to 17, wherein the data based on arrival direction information includes a report indicating at least one detected event and the direction of the detected event.

[0391] Example 19 includes a first device according to any one of Examples 1 to 18, and also includes a plurality of microphones.

[0392] Example 20 includes a first device according to any one of Examples 1 to 19, and further includes at least one speaker configured to output sound associated with at least one audio signal.

[0393] Example 21 includes a first device according to any one of Examples 1 to 20, wherein: the category corresponds to a classification of a specific sound represented by an audio signal and associated with a specific audio event; and the embedded part includes a signature or information corresponding to the specific sound or specific audio event, and is configured to detect the specific sound or specific audio event in other audio signals by processing other audio signals.

[0394] According to Example 22, a system includes: a first device according to any one of Examples 1 to 21; and a second device including: one or more processors configured to: receive data; and process the data to verify categories, modify audio data representing a sound scene based on direction of arrival information and embedding to generate modified audio data corresponding to the updated sound scene, or both.

[0395] According to Example 23, a system includes: a first device, including: a memory configured to store instructions; and one or more processors configured to: receive audio signals from a plurality of microphones; process the audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and transmit data based on the arrival direction information and a category associated with the arrival direction information; and a second device, including one or more processors, the one or more processors being configured to: receive data based on the arrival direction information and the category; obtain audio data and a category representing the sound associated with the arrival direction information; and verify the category based at least on the audio data and the arrival direction information.

[0396] According to Example 24, a system includes: a first device, including: a memory configured to store instructions; and one or more processors configured to: receive audio signals from a plurality of microphones; process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; and transmit data based on the direction-of-arrival information and embedded data associated with the direction-of-arrival information; and a second device, including one or more processors configured to: receive data based on the direction-of-arrival information and the embedded data; and process audio data representing a sound scene based on the direction-of-arrival information and the embedded data to generate modified audio data corresponding to an updated sound scene.

[0397] According to Example 25, a method of processing audio includes: receiving audio signals from a plurality of microphones at one or more processors of a first device; processing the audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and sending to a second device data based on the arrival direction information and category or embedded data associated with the arrival direction information.

[0398] Example 26 includes the method according to Example 25, and further includes processing signal data corresponding to the audio signal to determine the category or embedding.

[0399] Example 27 includes the method according to Example 26, and further includes performing beamforming operations on the audio signal to generate signal data.

[0400] Example 28 includes a method according to Example 26 or Example 27, wherein signal data is processed at one or more classifiers to determine a category from a plurality of categories supported by one or more classifiers for sounds represented in one or more audio signals and associated with audio events, and wherein the category is sent to a second device.

[0401] Example 29 includes a method according to any one of Examples 26 to 28, wherein signal data is processed at one or more encoders to produce an embedding corresponding to one or more sounds represented in an audio signal and associated with an audio event, and wherein the embedding is sent to a second device.

[0402] Example 30 includes the method according to any one of Examples 25 to 29, and further includes the representation of sending an audio signal to a second device.

[0403] Example 31 includes the method according to any one of Examples 25 to 30, further comprising: receiving data based on direction of arrival information and category at one or more processors of the second device; obtaining audio data and category representing a sound associated with the direction of arrival information at one or more processors of the second device; and verifying the category at one or more processors of the second device based at least on the audio data and the direction of arrival information.

[0404] Example 32 includes the method according to any one of Examples 25 to 31, further comprising: receiving data based on direction of arrival information and embedded data at one or more processors of the second device; and processing audio data representing a sound scene based on the direction of arrival information and embedded data at one or more processors of the second device to generate modified audio data corresponding to the updated sound scene.

[0405] According to Example 33, an apparatus includes: a memory configured to store instructions; and a processor configured to execute instructions to perform the method according to any one of Examples 25 to 30.

[0406] According to Example 34, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to perform the method according to any one of Examples 25 to 30.

[0407] According to Example 35, an apparatus includes components for implementing the method according to any one of Examples 25 to 30.

[0408] According to Example 36, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause the one or more processors to: receive audio signals from a plurality of microphones; process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented in one or more of the audio signals; and send to a second device data based on the direction-of-arrival information and category or embedded data associated with the direction-of-arrival information.

[0409] Example 37 includes a non-transitory computer-readable medium according to Example 36, wherein the instructions are executable to further cause one or more processors to send audio signals to a second device.

[0410] Example 38 includes a non-transitory computer-readable medium according to Example 37, wherein the representation of an audio signal corresponds to one or more beamforming audio signals.

[0411] According to Example 39, a first device includes: means for receiving audio signals from a plurality of microphones; means for processing the audio signals to generate arrival direction information corresponding to one or more sources of sound represented by one or more of the audio signals; and means for sending to a second device data based on the arrival direction information and category or embedded data associated with the arrival direction information.

[0412] This case includes the following third set of examples.

[0413] According to Example 1, a second device includes: a memory configured to store instructions; and one or more processors configured to: receive from the first device an indication of an audio category corresponding to an audio event.

[0414] Example 2 includes a second device according to Example 1, wherein one or more processors are further configured to: receive audio data representing a sound associated with an audio event from the first device; and process the audio data at one or more classifiers to verify that the sound corresponds to the audio event.

[0415] Example 3 includes a second device according to Example 2, wherein one or more processors are configured to provide an indication of audio data and audio category as input to one or more classifiers to determine a category associated with the audio data.

[0416] Example 4 includes a second device according to Example 2 or Example 3, wherein the audio category corresponds to a vehicle event, and wherein one or more processors are further configured to send a notification of the vehicle event to one or more third devices based on the location of the first device and the location of one or more third devices.

[0417] Example 5 includes a second device according to any one of Examples 2 to 4, wherein one or more processors are further configured to send control signals to the first device based on the output of one or more classifiers.

[0418] Example 6 includes a second device according to Example 5, wherein a control signal instructs the first device to perform an audio scaling operation.

[0419] Example 7 includes a second device according to Example 5 or Example 6, wherein a control signal instructs a first device to perform spatial processing based on the direction of a sound source.

[0420] Example 8 includes a second device according to any one of Examples 2 to 7, wherein one or more processors are further configured to: receive directional data corresponding to a sound source from a first device; and provide audio data, directional data, and an indication of audio category as input to one or more classifiers to determine a category associated with the audio data.

[0421] Example 9 includes a second device according to any one of Examples 2 to 8, wherein the audio data includes one or more beamforming signals.

[0422] Example 10 includes a second device according to any one of Examples 1 to 9, wherein one or more processors are further configured to: receive from the first device directional data corresponding to sound sources associated with an audio event; update a map of directional sound sources in an audio scene based on the audio event to generate an updated map; and send data corresponding to the updated map to one or more third devices geographically distant from the first device.

[0423] Example 11 includes a second device according to any one of Examples 1 to 10, wherein memory and one or more processors are integrated into a mobile phone, and wherein the first device corresponds to a headset device.

[0424] Example 12 includes a second device according to any one of Examples 1 to 10, wherein memory and one or more processors are integrated into a vehicle.

[0425] Example 13 includes a second device according to any one of Examples 1 to 12, and further includes a modem, wherein the indication of the audio type is received via the modem.

[0426] Example 14 includes a second device according to any one of Examples 1 to 13, wherein one or more processors are configured to selectively bypass the arrival direction processing of received audio data corresponding to an audio event based on whether arrival direction information is received from the first device.

[0427] Example 15 includes a second device according to any one of Examples 1 to 14, wherein one or more processors are configured to selectively bypass beamforming operations based on whether the received audio data corresponds to a multi-channel microphone signal from the first device or to a beamforming signal from the first device.

[0428] Example 16 includes a second device according to any one of Examples 1 to 15, wherein: the audio category corresponds to a classification of a specific sound represented by an audio signal and associated with an audio event.

[0429] According to Example 17, a system includes: a second device according to any one of Examples 1 to 16; and a first device including: one or more processors configured to: receive audio signals from one or more microphones; process the audio signals to determine an audio category; and send an indication of the audio category to the second device.

[0430] According to Example 18, a system includes: a first device including: one or more processors configured to: receive audio signals from one or more microphones; process the audio signals to determine an audio category corresponding to an audio event; and send an indication of the audio category; and a second device including one or more processors configured to: receive an indication of an audio category corresponding to an audio event.

[0431] According to Example 19, a method includes: receiving an indication of an audio category at one or more processors of a second device, the indication being received from a first device and corresponding to an audio event; and processing audio data at one or more processors of the second device to verify that a sound represented by the audio data corresponds to the audio event.

[0432] Example 20 includes the method according to Example 19, further including receiving audio data from a first device, and wherein processing of the audio data includes providing the audio data as input to one or more classifiers to determine a classification associated with the audio data.

[0433] Example 21 includes the method according to Example 20, wherein the processing of audio data further includes providing an indication of the audio category as a second input to one or more classifiers to determine the category associated with the audio data.

[0434] Example 22 includes the method according to Example 20 or Example 21, and further includes sending control signals to a first device based on the outputs of one or more classifiers.

[0435] Example 23 includes the method according to Example 22, wherein the control signal includes an audio scaling instruction.

[0436] Example 24 includes a method according to Example 22 or Example 23, wherein the control signal includes instructions for performing spatial processing based on the direction of the sound source.

[0437] Example 25 includes the method according to any one of Examples 19 to 24, wherein the audio category corresponds to a vehicle event, and the method further includes sending a notification of the vehicle event to one or more third devices based on the location of a first device and the location of one or more third devices.

[0438] Example 26 includes the method according to any one of Examples 19 to 25, further comprising: receiving from a first device directional data corresponding to a sound source associated with an audio event; updating a map of directional sound sources in an audio scene based on the audio event to generate an updated map; and sending data corresponding to the updated map to one or more third devices geographically distant from the first device.

[0439] Example 27 includes the method according to any one of Examples 19 to 26, further comprising selectively bypassing the arrival direction processing of received audio data corresponding to an audio event based on whether arrival direction information is received from the first device.

[0440] Example 28 includes the method according to any one of Examples 19 to 27, further comprising selectively bypassing beamforming operations based on whether the received audio data corresponds to a multi-channel microphone signal from the first device or to a beamforming signal from the first device.

[0441] Example 29 includes the method according to any one of Examples 19 to 28, further comprising: receiving an audio signal from one or more microphones at one or more processors of the first device; processing the audio signal at one or more processors of the first device to determine an audio category; and sending an indication of the audio category from the first device to a second device.

[0442] According to Example 30, an apparatus includes: a memory configured to store instructions; and a processor configured to execute instructions to perform a method according to any one of Examples 16 to 28.

[0443] According to Example 31, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause one or more processors to perform the method according to any one of Examples 16 to 29.

[0444] According to Example 32, an apparatus includes components for performing the method according to any one of Examples 16 to 28.

[0445] According to Example 33, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a second device, cause one or more processors to receive from a first device an indication of an audio category corresponding to an audio event.

[0446] Example 34 includes a non-transitory computer-readable medium according to Example 33, wherein the instructions are executable to further cause one or more processors to: receive audio data representing a sound associated with an audio event from a first device; and process the audio data at one or more classifiers to verify that the sound corresponds to the audio event.

[0447] Example 35 includes a non-transitory computer-readable medium according to Example 34, wherein the instructions are executable to further cause one or more processors to provide an indication of audio data and audio category as input to one or more classifiers to determine the category associated with the audio data.

[0448] Example 36 includes a non-transitory computer-readable medium according to Example 34 or Example 35, wherein the instructions are executable to further cause one or more processors to: receive direction data corresponding to a sound source from a first device; and provide the audio data, direction data, and an indication of audio category as input to one or more classifiers to determine a category associated with the audio data.

[0449] According to Example 37, an apparatus includes: a component for receiving an indication of an audio category, the indication being received from a remote device and corresponding to an audio event; and a component for processing audio data to verify that a sound represented by the audio data corresponds to the audio event.

[0450] According to Example 38, a second device includes: a memory configured to store instructions; and one or more processors configured to: receive from a first device: audio data representing sound; and an indication that the audio data corresponds to an audio category associated with a vehicle event; process the audio data at one or more classifiers to verify that the sound represented by the audio data corresponds to a vehicle event; and send a notification of the vehicle event to one or more third devices based on the location of the first device and the location of one or more third devices.

[0451] According to Example 39, a method includes: receiving audio data from a first device and an indication that the audio data from the first device corresponds to an audio category associated with a vehicle event at one or more processors of a second device; processing the audio data at one or more classifiers of the second device to verify that the sound represented by the audio data corresponds to a vehicle event; and sending a notification of the vehicle event to one or more third devices based on the location of the first device and the location of one or more third devices.

[0452] According to Example 40, a device includes: a memory configured to store instructions; and a processor configured to execute instructions to perform the method according to Example 39.

[0453] According to Example 41, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a second device, cause one or more processors to perform the method according to Example 39.

[0454] According to Example 42, an apparatus includes a component for performing the method according to Example 39.

[0455] According to Example 43, a first device includes: a memory configured to store instructions; and one or more processors configured to: receive one or more audio signals from one or more microphones; process the one or more audio signals to determine whether a sound represented by one or more of the audio signals originates from an identifiable direction; and selectively send arrival direction information of the sound source to a second device based on the determination.

[0456] According to Example 44, a method includes: receiving one or more audio signals from one or more microphones at one or more processors of a first device; processing the one or more audio signals at one or more processors to determine whether a sound represented by one or more of the audio signals originates from an identifiable direction; and selectively sending arrival direction information of the sound source to a second device based on the determination.

[0457] According to Example 45, a device includes: a memory configured to store instructions; and a processor configured to execute instructions to perform the method according to Example 44.

[0458] According to Example 46, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause one or more processors to perform the method according to Example 44.

[0459] According to Example 47, an apparatus includes a component for performing the method according to Example 44.

[0460] According to Example 48, a first device includes: a memory configured to store instructions; and one or more processors configured to: receive one or more audio signals from one or more microphones; determine, based on one or more criteria, whether to send one or more audio signals to a second device or to send beamformed audio signals generated based on one or more audio signals to the second device; and based on the decision, send audio data corresponding to one or more audio signals or corresponding to beamformed audio signals to the second device.

[0461] According to Example 49, a method includes: receiving one or more audio signals from one or more microphones at one or more processors of a first device; determining at one or more processors, based on one or more criteria, whether to send one or more audio signals to a second device or to send beamforming audio signals generated based on one or more audio signals to the second device; and sending audio data corresponding to or corresponding to the one or more audio signals to the second device based on the decision.

[0462] According to Example 50, a device includes: a memory configured to store instructions; and a processor configured to execute instructions to perform the method according to Example 49.

[0463] According to Example 51, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a first device, cause one or more processors to perform the method according to Example 49.

[0464] According to Example 52, an apparatus includes a component for performing the method according to Example 49.

[0465] According to Example 53, a second device includes: a memory configured to store instructions; and one or more processors configured to: receive from a first device: audio data representing sound; directional data corresponding to a sound source; and a classification of sound corresponding to an audio event; process the audio data to verify that the sound corresponds to the audio event; update a map of directional sound sources in an audio scene based on the audio event to generate an updated map; and send data corresponding to the updated map to one or more third devices geographically distant from the first device.

[0466] According to Example 54, a method includes: receiving, at one or more processors of a second device, audio data representing sound, directional data corresponding to a sound source, and a classification of sound corresponding to an audio event, the audio data, directional data, and classification being received from a first device; processing, at one or more processors, the audio data to verify that the sound corresponds to the audio event; updating, at one or more processors, a map of directional sound sources in an audio scene based on the audio event to generate an updated map; and sending, at one or more third devices geographically distant from the first device, data corresponding to the updated map.

[0467] According to Example 55, a device includes: a memory configured to store instructions; and a processor configured to execute instructions to perform the method according to Example 54.

[0468] According to Example 56, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors of a second device, cause one or more processors to perform the method according to Example 54.

[0469] According to Example 57, an apparatus includes a component for performing the method according to Example 54.

[0470] Those skilled in the art will further understand that the various illustrative logic blocks, configurations, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of both. The various illustrative elements, blocks, configurations, modules, circuits, and steps have been described above in general terms of their function. Whether this function is implemented as hardware or processor-executable instructions depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, and such implementation decisions should not be construed as departing from the scope of this document.

[0471] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be directly embodied in hardware, a software module executed by a processor, or a combination of both. The software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, optical disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor, allowing the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be integrated into the processor. The processor and storage medium can reside in a dedicated integrated circuit (ASIC). The ASIC can reside in a computing device or user terminal. Alternatively, the processor and storage medium can reside as separate components in a computing device or user terminal.

[0472] The prior description of the disclosed forms is provided to enable those skilled in the art to manufacture or use the disclosed forms. Those skilled in the art will readily understand various modifications to these forms, and the principles defined herein can be applied to other forms without departing from the scope of this application. Therefore, this application is not intended to be limited to the forms shown herein, but rather to conform to the widest possible scope consistent with the principles and novel features defined by the appended claims. [Simplified Explanation of the Diagram]

[0011] Figure 1 is a block diagram of a particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0012] Figure 2 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0013] Figure 3 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0014] Figure 4 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0015] Figure 5 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0016] Figure 6 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0017] Figure 7 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0018] Figure 8 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0019] Figure 9 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0020] Figure 10 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0021] Figure 11 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case, and includes a graphical illustration of audio content separation.

[0022] Figure 12 is a diagram of a specific implementation of an operation that can be performed in an audio processing device according to some examples of this case.

[0023] Figure 13 is a diagram of another specific implementation of an operation that can be performed in an audio processing device according to some examples of this case.

[0024] Figure 14 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0025] Figure 15 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0026] Figure 16 is a block diagram of another particular illustrative example of a system configured to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case.

[0027] Figure 17 illustrates an example of an audio scene according to some examples of this case, including multiple directional sound sources that can be determined by directional processing of one or more audio signals received from one or more microphones.

[0028] Figure 18 illustrates an example of a shared audio scenario involving multiple directional sound sources, based on some examples of this case.

[0029] Figure 19 illustrates an example of an integrated circuit including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0030] Figure 20 is a diagram of an active device including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0031] Figure 21 is a diagram of an earphone including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0032] Figure 22 is a diagram of a wearable electronic device including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0033] Figure 23 is a diagram of a volume-controlled loudspeaker system including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0034] Figure 24 is a diagram of a camera including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0035] Figure 25 is a diagram of an earphone (such as a virtual reality, mixed reality, or augmented reality earphone) including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0036] Figure 26 is a diagram of a mixed reality or augmented reality glasses device including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0037] Figure 27 is a diagram of an earphone including a directional audio signal processing unit for generating directional audio signal data, according to some examples of this case.

[0038] Figure 28 is a diagram of a first example of a vehicle including a directional audio signal processing unit for navigating the vehicle, according to some examples of this case.

[0039] Figure 29 is a diagram of a second example of a vehicle including a directional audio signal processing unit for navigating the vehicle, according to some examples of this case.

[0040] Figure 30 is a diagram of a specific embodiment of a method for processing audio messages according to some examples of this case.

[0041] Figure 31 is a diagram of another specific embodiment of a method for processing audio according to some examples of this case.

[0042] Figure 32 is a diagram of another specific embodiment of a method for processing audio according to some examples of this case.

[0043] Figure 33 is a diagram of another specific embodiment of a method for processing audio according to some examples of this case.

[0044] Figure 34 is a diagram of another specific embodiment of a method for processing audio according to some examples of this case.

[0045] Figure 35 is a diagram of another specific embodiment of a method for processing audio according to some examples of this case.

[0046] Figure 36 is a diagram of another specific embodiment of a method for processing audio according to some examples of this case.

[0047] Figure 37 is a block diagram of a particular illustrative example of a device operable to perform directional processing on one or more audio signals received from one or more microphones, according to some examples of this case. [Biomaterial Storage]

[0474] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A first device, comprising: One memory module is configured to store data; and one or more processors coupled to the memory, wherein the one or more processors are configured to: receive audio signals from a plurality of microphones; process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; send the data based on the direction-of-arrival information and a category or embedding associated with the direction-of-arrival information to a second device; and selectively send a representation of the audio signals to the second device according to an available power level at the first device.

2. The first device according to claim 1, wherein the one or more processors are further configured to process signal data corresponding to the audio signals to determine the category or embedding.

3. The first device according to claim 2, wherein the one or more processors are further configured to perform a beamforming operation on the audio signals to generate the signal data.

4. The first device according to claim 2, wherein the one or more processors are further configured to process the signal data at one or more classifiers to determine the category from a plurality of categories supported by the one or more classifiers for a sound represented by one or more of the audio signal and associated with an audio event, and wherein the category is sent to the second device.

5. The first device according to claim 2, wherein the one or more processors are further configured to process the signal data at one or more encoders to generate the embedding, the embedding corresponding to a sound represented in one or more of the audio signal and associated with an audio event, and wherein the embedding is sent to the second device.

6. The first device according to claim 1, wherein the one or more processors are further configured to process image data at one or more encoders to generate the embedding, the embedding corresponding to an object represented by the image data and associated with an audio event, and wherein the embedding is sent to the second device.

7. The first device according to claim 6 further includes one or more cameras configured to generate the image data.

8. The first device according to request item 1, wherein: The category corresponds to a classification of a specific sound represented by those audio signals and associated with a specific audio event; and the embedding includes a signature or information corresponding to the specific sound or specific audio event, and is configured to detect the specific sound or specific audio event in other audio signals by processing other audio signals.

9. The first device according to claim 1, wherein the one or more processors are further configured to: perform spatial processing on the audio signals based on the direction of arrival information to generate one or more beamforming audio signals, wherein the representation of the audio signals corresponds to the one or more beamforming audio signals.

10. The first device according to claim 1, wherein the memory and the one or more processors are integrated into a headset device, and wherein the second device corresponds to a mobile phone.

11. The first device according to claim 1 further includes a modem, wherein the data is transmitted to the second device via the modem.

12. The first device according to claim 11, wherein the representation of the audio signals corresponds to the audio signals, one or more beamforming audio signals, or a simplified version of the audio signals.

13. The first device according to request item 1, wherein the one or more processors are further configured to generate a user interface output indicating at least one of an environmental event or an acoustic event.

14. The first device according to claim 1, wherein the one or more processors are further configured to receive data from the second device indicating an acoustic event, the acoustic event being associated with a sound represented in one or more of the audio signals.

15. The first device according to claim 1, wherein the one or more processors are further configured to: receive direction information associated with the audio signal from the second device; and perform an audio scaling operation based on the direction information.

16. The first device according to claim 1, wherein the one or more processors are integrated in a vehicle.

17. The first device according to claim 1, wherein the data based on the direction of arrival information includes a report indicating at least one detected event and the direction of the detected event.

18. The first device according to claim 1 also includes a plurality of microphones.

19. The first device according to claim 1 further includes at least one speaker configured to output a sound associated with at least one of the audio signals.

20. The first device according to claim 1, wherein, in order to selectively transmit the representation of the audio signals, the one or more processors are further configured to: compare the available power amount with a power threshold; and transmit the representation of the audio signals to the second device if the available power amount is greater than or equal to the power threshold.

21. The first device according to claim 1, wherein, in order to selectively transmit the representation of the audio signals, the one or more processors are further configured to: compare the available power amount with a power threshold; determine whether beamforming audio signals are available if the available power amount is less than the power threshold; and determine not to transmit the representation of the audio signals to the second device if the beamforming audio signals are unavailable.

22. The first device according to claim 1, wherein, in order to selectively transmit the representation of the audio signals, the one or more processors are further configured to: compare the available power amount with a power threshold; determine whether beamforming audio signals are available if the available power amount is less than the power threshold; and transmit the representation of the audio signals to the second device if the beamforming audio signals are available, wherein the representation of the audio signals corresponds to the beamforming audio signals.

23. A method for processing audio messages, the method comprising: At one or more processors in a first device, audio signals from multiple microphones are received; the audio signals are processed to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; the direction-of-arrival information and a category or embedded data associated with the direction-of-arrival information are sent to a second device; and a representation of the audio signals is selectively sent to the second device according to an available power level at the first device.

24. The method according to claim 23 further includes processing signal data corresponding to the audio signals to determine the category or embedding; and performing beamforming operations on the audio signals to generate the signal data.

25. The method according to claim 23, further comprising the steps of: processing signal data corresponding to the audio signals to determine the category or embedding, wherein the signal data is processed at one or more classifiers to determine the category from a plurality of categories supported by the one or more classifiers for sounds represented by one or more of the audio signals and associated with an audio event, and wherein the category is sent to the second device.

26. The method of claim 23, wherein the data is sent to the second device to enable distributed audio processing at a distributed audio system, the distributed audio system including the first device and the second device, the distributed audio system including one or more low-complexity audio processing jobs associated with the first device.

27. The method according to request item 23 further includes: At one or more processors of the second device, data based on the direction of arrival information and the category is received; At one or more processors of the second device, audio data representing the sound associated with the direction of arrival information and the category are obtained; and at one or more processors of the second device, the category is verified at least based on the audio data and the direction of arrival information.

28. The method according to request item 23 also includes: The arrival direction information and the embedded data are received at one or more processors of the second device; And at one or more processors of the second device, audio data representing a sound scene is processed based on the direction of arrival information and the embedding to generate modified audio data corresponding to the updated sound scene.

29. A non-transitory computer-readable medium including instructions that, when executed by one or more processors of a first device, cause the one or more processors to: receive audio signals from a plurality of microphones; process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented by one or more of the audio signals; transmit to a second device a representation of the audio signals based on the direction-of-arrival information and a category or embedded data associated with the direction-of-arrival information; and selectively transmit to the second device a representation of the audio signals according to an available power level.

30. A first device, comprising: A component used to receive audio signals from multiple microphones; A means for processing the audio signals to generate arrival direction information corresponding to one or more sources of sound represented in one or more of the audio signals; a means for transmitting to a second device data based on the arrival direction information and a category or embedded data associated with the arrival direction information; and a means for selectively transmitting a representation of the audio signals to the second device according to an available power level.

Images