Hearing augmentation and wearable system with localized feedback

Abstract

To provide techniques including devices and system implementing the techniques to provide a user with feedback of an event when the user is wearing a wearable device.SOLUTION: The techniques include measuring ambient sound using two or more microphones on the wearable device. The measured ambient sound is used to determine a related event worth relaying to the user. Based on the location attribute and sound properties, the nature and / or class of the event may be ascertained using pattern recognition algorithms according to user threshold settings. Insignificant events that the user prefers to ignore will be ruled out by the algorithm. Upon determining the event that merits the user's attention, the wearable device provides the user with feedback indicating the nature and location of the event.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Patent Application No. 17 / 027,919, filed September 22, 2020, which is assigned to the assignee hereof and is hereby incorporated by reference in its entirety as if fully set forth below and for all applicable purposes.

[0002] FIELD OF THEINVENTION Aspects of the present disclosure relate generally to wireless communications, and more specifically to audio signal processing involving a wearable device in wireless communication with a computing device. [Background technology]

[0003] Wearable devices can provide the user with a desired transmitted or reproduced audio experience by masking, guarding against, or canceling ambient noise. High volume output or white noise generated by the wearable device can mask ambient noise. Soundproofing reduces sound pressure by reflecting or absorbing sound energy. Noise cancellation, or active noise control / reduction, reduces ambient noise by the addition of a second sound that cancels the ambient noise. In these cases, the user may fail to pay attention to events in the environment that require attention, such as alarms, alarm sounds, or verbal communication. Furthermore, when the user is focused on the task at hand, the user may lack other senses (e.g., vision) that would otherwise be available to alert the user about such events.

[0004] Therefore, methods for providing users with appropriate and non-intrusive (e.g., preserving the desired audio experience) notifications regarding certain background events, as well as devices and systems configured to implement these methods, are desired. Summary of the Invention

[0005] All examples and features mentioned in this specification can be combined in any technically possible manner.

[0006] Aspects of the present disclosure provide a method for providing feedback to a user of a wearable device, the method including measuring ambient sound using two or more microphones on the wearable device, determining an event based on the measured ambient sound, determining a location attribute of the event relative to the wearable device, and providing feedback to the user based on the event and the location attribute.

[0007] In an aspect, providing feedback to the user based on the event and location attributes includes playing spatialized audio cues in addition to the audio content.

[0008] In an aspect, determining the event based on the measured ambient sound includes measuring a sound level loudness of the event and determining a location of the event relative to the wearable device using an audio signal associated with the event, the location of the event being determined based on a direction and distance processed from the audio signal, and the location attributes of the event include at least one of a direction or a distance. In some cases, determining the event based on the ambient sound further includes filtering out incidents of the event based on at least one of a location of the event relative to the wearable device, a repetitive pattern in the audio signal associated with the event, or attributes of the audio signal associated with the event.

[0009] In aspects, determining the event based on the measured ambient sound includes determining that a sound level magnitude of the event exceeds a threshold corresponding to a location of the event relative to the wearable device. In some cases, the threshold varies based on the location of the event relative to the wearable device, and the threshold increases as the location of the event moves closer to the user.

[0010] In aspects, determining the event based on the ambient sound is based at least in part on correlating i) a location of the event relative to the wearable device, ii) a repeating pattern in the audio signal associated with the event, and iii) attributes of the audio signal associated with the event. In some cases, determining the event is based at least in part on deep learning analysis performed by artificial intelligence using data of the location of the event relative to the wearable device, the repeating pattern in the audio signal associated with the event, and the attributes of the audio signal associated with the event.

[0011] In an aspect, providing feedback to the user based on the event includes one or more of playing spatialized sound using at least one speaker of the wearable device, providing directional tactile notifications using the wearable device, or displaying notifications on one or more devices connected to the wearable device.

[0012] In some cases, the spatialized sound represents a location of the determined event relative to the user, the represented location being based at least on the location attribute.

[0013] In an aspect, the method further includes determining that the user is in a state of concentration based on a pattern of motion signals measured by the wearable device, and providing feedback to the user based on the event and location attributes is based on the determined state of concentration. In some cases, determining that the user is in a state of concentration includes processing a pattern of motion signals measured by one or more inertia measurement units (IMUs) disposed on the wearable device.

[0014] In an embodiment, the two or more microphones on the wearable device include at least three microphones positioned to measure ambient sounds surrounding the user.

[0015] In aspects, the wearable device is an audio device worn on at least one of the user's head or neck.

[0016] In aspects, the wearable device includes at least one of over-ear headphones, on-ear headphones, in-ear headphones, earphones, completely wireless earphones, glasses, a headband, a neckband, an ear hook, an ear clip, a head-worn audio device, or a neck-worn audio device.

[0017] In an aspect, the wearable device is a noise cancelling audio device.

[0018] Aspects of the present disclosure provide a system for providing user spatialized feedback, the system including a wearable device having two or more microphones configured to measure ambient sound, and a computing device coupled to the wearable device, the computing device configured to determine an event based on the measured ambient sound, determine a location attribute of the event relative to the wearable device, and provide feedback to a user based on the event and the location attribute.

[0019] In an aspect, the wearable device further includes two or more speakers configured to play audio content that cancels the measured ambient sounds and to alert the user to the determined event by playing spatialized audio cues in addition to the audio content.

[0020] In an aspect, the wearable device further includes an actuator for providing haptic feedback in response to the computing device providing feedback to the user based on the event.

[0021] In an aspect, the computing is further configured to exclude incidents of the event based on at least one of a location of the event relative to the wearable device, a repetitive pattern in an audio signal associated with the event, or an attribute of the audio signal associated with the event.

[0022] An aspect of the disclosure provides a method for providing feedback to a user of a wearable device, the method including measuring ambient sound using two or more microphones on the wearable device, determining an event based on the measured ambient sound exceeding a threshold that varies over two or more different constituent events corresponding to two or more different situational criteria, and providing feedback to the user based on the determined event.

[0023] In an aspect, each of the two or more different situation criteria includes a customization configuration for at least one of a threshold associated with the determined event, a location attribute of the event relative to the wearable device, or feedback to the user based on the determined event.

[0024] In an aspect, the customization configuration is input by a user.

[0025] In an aspect, the method further includes associating the customization configuration with location attributes of the determined event for the wearable device, and identifying patterns to determine general rules for the configuration based on the location attributes based on the user input and association.

[0026] In an aspect, the customized configuration includes a default factory configuration.

[0027] In an aspect, each of the two or more different situation criteria includes a location attribute for the wearable device, the location attribute being determined based on ambient sound measured using two or more microphones of the wearable device.

[0028] In aspects, the threshold comprises a minimum or maximum distance corresponding to one of two or more different situation criteria.

[0029] In an embodiment, the threshold comprises a direction corresponding to one of two or more different situation criteria.

[0030] In an embodiment, each of the two or more different situation criteria corresponds to a physical location of the wearable device.

[0031] In an embodiment, each of the two or more different situation criteria includes a sound class including at least one of speech, background sounds, or music sequences.

[0032] In aspects, the background sounds include alarms, footsteps, traffic noise, or sound patterns.

[0033] In an embodiment, the threshold comprises a minimum sound level that corresponds to one of two or more different situational criteria.

[0034] In an embodiment, the threshold value corresponds to a sensitivity to one of two or more different situation criteria.

[0035] In an aspect, providing feedback includes playing a recorded copy of the ambient sound to the user.

[0036] In aspects, playing the recorded copy includes amplifying or attenuating the recorded copy of the ambient sound based on data collected by a feedback microphone in the wearable device.

[0037] In aspects, amplifying or attenuating the recorded copy of the ambient sound is triggered by movement of the wearable device.

[0038] In an aspect, playing the recorded copy includes delaying playing the recorded copy based on movement of the wearable device.

[0039] In an aspect, the method further includes correlating ambient sounds with location attributes over a period of time, generating a saliency map that maps variations in ambient sounds over the period at different locations, measuring a movement record of the wearable device over the period of time, and determining a user response based on a relationship between the movement record and the saliency map.

[0040] In an aspect, determining the event further includes transmitting the measured ambient sound to a second device for processing and receiving the determined event and corresponding feedback for providing to the user from the second device, the second device having greater processing capabilities than the wearable device.

[0041] Two or more features described in this disclosure, including features described in this Summary section, may be combined to form implementations not specifically described herein.

[0042] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0043] [Figure 1] 1 illustrates an example of a system in which aspects of the present disclosure may be implemented. [Figure 2A] 1 illustrates an exemplary wireless audio device in accordance with certain aspects of the present disclosure. [Figure 2B] 1 illustrates an exemplary computing device in accordance with certain aspects of the present disclosure. [Diagram 3] 13 illustrates example operations for providing feedback of determined events and location attributes to a user in accordance with certain aspects of the present disclosure. [Figure 4] 1 illustrates an example of data processing for extracting location attributes of an event, according to certain aspects of the present disclosure. [Diagram 5]1 illustrates an example of a visual presentation of determined events and associated location attributes in accordance with certain aspects of the present disclosure. [Figure 6A] 1 illustrates an example process for determining an event requiring user attention, according to certain aspects of the present disclosure. [Figure 6B] 1 illustrates an example of a deep learning process for determining an event requiring a user's attention, according to certain aspects of the present disclosure. [Figure 6C] 1 illustrates an example of a deep learning process for determining an event, according to certain aspects of the present disclosure. [Figure 7] 1 illustrates an example process for determining an event based on sound energy, according to certain aspects of the present disclosure. [Figure 8] 1 illustrates movement data used to determine a user's state, according to certain aspects of the present disclosure. [Figure 9] 13 illustrates example operations for providing feedback of determined events and location attributes to a user in accordance with certain aspects of the present disclosure. [Figure 10] 1 illustrates an example interface for adjusting customization configurations of situation criteria, according to certain aspects of the present disclosure. [Figure 11] 1 illustrates an example interface for adjusting customization configurations of situation criteria, according to certain aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] Like numbers refer to like elements.

[0045] The present disclosure provides processes, methods, systems, and devices for providing feedback of events (e.g., interrupting events, or events that require the user's attention and are not events that the user ignores) to a user while the user is wearing a wearable audio output device. For example, without feedback, the user may not pay attention to the event due to enjoying the audio experience provided by the wearable device, such as when the wearable device cancels out ambient noise to allow the user to be fully immersed in the audio experience. According to the present disclosure, the wearable device can determine the event based on certain conditions, such as sound location relative to the user, sound class (e.g., nature of the detected sound, such as speech, alarm, etc.), environment (e.g., home, office, or specific room, public, etc.), and other conditions defined by customized parameters.

[0046] In an aspect, the method includes measuring ambient sounds (i.e., sounds that are not part of the audio experience) using two or more microphones on the wearable device. An event is determined based on the measured ambient sounds. A location attribute of the event is determined (this may be referred to as "localization"). Feedback of the determined event and location attributes is then provided to the user. In some cases, determining the event based on the measured ambient sounds uses a deep learning based algorithm that identifies and removes occurrences in the surrounding environment that do not require the user's attention. The algorithm may determine the event using various characteristics of sounds in loudness, location, and ambient sounds. Feedback of the determined event may be presented to the user, the feedback including, for example, location attributes and / or informing the user where attention should be directed.

[0047] In an aspect, an event is determined based on the measured ambient sound exceeding a threshold. The threshold may vary for two or more different constituent events corresponding to two or more different situation criteria. The two or more different situation criteria may include at least one of a threshold associated with the determined event, a location attribute of the event relative to the wearable device, or feedback to the user based on the determined event. For example, an event including speech and an event including a door knock may be configured with different thresholds for detection. Similarly, an event 5 meters away and an event 10 meters away may also be configured with different thresholds for detection. The threshold level may also be customized based on the location (e.g., relative distance and direction) of the event relative to the wearable device. The user may also customize the form and characteristics (e.g., brightness, volume, and / or magnitude) of the feedback, such as visual, auditory, combinations thereof, and other forms.

[0048] Users often lose at least some situational awareness when using wearable audio devices, especially when using active noise canceling headphones that are intended to remove ambient sound. In another example, situational awareness is reduced when the volume of the audio is at an excessive level that masks ambient sound, or when the user has good sound insulation (e.g., passive sound insulation). Another example of situational awareness is reduced when the user is in a state of concentration, such as when working, studying, or reading, with the aid of a wearable audio device (e.g., canceling or attenuating ambient sound). Reduced situational awareness can have costly or bothersome consequences, such as when an event merits a timely response. The present disclosure provides techniques for providing a user with spatialized feedback of events that may merit attention, while providing the user with an enjoyable audio experience.

[0049] In aspects, the disclosed techniques can enhance a user's awareness using sound event detection and audio cues representing the detected sound event. In an office or telecommuting environment, a user may be in a focused state and less aware of the user's surroundings. A noise-canceling wearable device may further reduce the user's awareness. The user may want or need to pay attention to an incoming verbal communication, a doorbell, or other alarm or sound notification that can be effectively canceled by the wearable device. In this way, the disclosed techniques allow the user to focus on and benefit from a noise-canceling wearable device without losing situational awareness. In some cases, the audio cues are designed to be informative and minimally disruptive. This allows the user to determine when and how to react to an event after learning about it, while still maintaining the option to ignore the event. In aspects, additional visualization or notification strategies are used along with the audio cues to help the user interpret the event.

[0050] The disclosed technique is implemented using a combination of a microphone and an inertial measurement unit (IMU). For example, the technique first sets a loudness threshold to detect an intrusive event that is different from base level noise. This step may be referred to as "event detection". The technique then determines one or more location attributes of the intrusive event, such as identifying the direction and distance of the event. The technique then uses input from the IMU to determine whether the user may have caused the detected event or is already aware of the detected event. If the event is determined to merit the user's attention, the technique further includes providing spatialized audio feedback to the user to inform the user of the location of the event.

[0051] In aspects, in addition to the location attribute of the interruption event, different thresholds may be defined or configured for different configuration events. The configuration events may be defined by corresponding situational criteria such as decibel level (e.g., event threshold), characteristics of the ambient sound (e.g., sound class or type, such as speech, alarm, melody, etc.), location relative to the user (e.g., distance and direction), etc. The situational criteria may be provided by the user or may be configured using a default factory or production profile or configuration. For example, the user may provide input to customize the customization configuration via a graphical interface on the wearable device or on a computing device in communication with the wearable device. Examples are further discussed in connection with FIGS. 9-10.

[0052] In an aspect, localization and event detection are performed on the wearable device to minimize latency and provide access to the full array of microphones on the wearable device. The user may be prompted with audio cues spatialized to the direction of the event (e.g., left or right, up or down) when the following example operations are performed. First, an interval detection algorithm is executed using the microphones on the wearable device and connected / controlled by a processor (either on the wearable device or on an external computing device). As mentioned, the interval detection algorithm may be executed on the wearable device to minimize transmission latency. In some cases, the interval detection algorithm may be executed on a computing device that wirelessly communicates with the wearable device to use a more powerful processor in the computing device. The interval detection algorithm detects sudden changes in sound energy (e.g., sudden increases in decibel levels) to identify events distinct from ambient sounds. The interval detection algorithm detects characteristics of sounds, such as speech or voice intervals, to identify events distinct from ambient sounds.

[0053] Second, location attributes of the event are determined and a second layer of data processing confirms the nature of the event. For example, by determining the distance of the event, the interval detection algorithm can further distinguish sounds made by the user (e.g., the user's own speech) from sounds of events in the far field (e.g., door knocks across the room). In some cases, events that are determined to be facing the user (i.e., in front of the user within a certain distance) are ignored because such events may be visually acquired by the user. Location attributes may include distance, direction, or both. Distance and direction may be calculated using audio data captured by two or more microphones spaced apart within the wearable device.

[0054] Ascertaining the nature of the event may also include processing the sound to determine a sound class and / or sound characteristics. For example, different constituent events may correspond to different sound classes, such as speech, alarms, or loud background events (e.g., knocking, crying, etc.). The threshold for each event of a particular nature may be adjusted or personalized using different sensitivity settings and thus may depend on the user's environment and preferences.

[0055] Third, the movement measurements are correlated with the event to determine whether the user needs to be notified about the event. For example, the lack of movement measured by the motion sensor may indicate that the user is in a state of concentration. The user may be more likely to desire event feedback while in a state of concentration. In addition, the movement measurements can help the interval detection algorithm to correctly identify the location attribute, taking into account the relative location fluctuations caused by the user's own movement.

[0056] In some cases, the movement measurements can form a pattern associated with the customization configuration and location attributes of the determined event. For example, a user can consistently / habitually respond to an event by turning in a certain direction. Such a pattern of movement is then correlated with the event. The event can be detected by associating the customization configuration with location attributes for the wearable device, such as by specifying a threshold based on a sound class associated with where the sound comes from. In one example, when a user wears the wearable device in a certain position facing toward a certain direction, a siren sound coming through a window can be noise-cancelled and ignored (i.e., not classified as an event for feedback). In such a case, random movements can be recorded when the siren is measured. Based on the random movements, a general rule can be determined for ignoring such siren sounds coming from the same direction. In comparison, a knock sound coming through a door can frequently cause a user to turn in a certain direction. Such a pattern of responses can be recorded and identified to determine a general rule of configuration, such as to provide feedback (e.g., an audio notification in the wearable device while the knock sound is noise-cancelled) when an event that fits the identified pattern is detected.

[0057] In aspects, the interval detection algorithm may include a lightweight threshold-based digital signal processing (DSP) algorithm that runs on a processor embedded in the wearable device. The wearable device may provide a user interface, such as a sensitivity slider, to control a threshold that adjusts the detection sensitivity. For example, a more sensitive setting may result in reporting events with less scrutiny, resulting in an increase in false positive detections. In addition to providing spatialized audio cues to the user, the wearable device may further provide information to an external computing device. The external computing device may display the event and its location attributes on a user interface. In one example, the user interface shows a "radar" display that informs the user about the nature of the event (e.g., type of event, such as speech, knocking, alarm, etc.) and the location of the event relative to the user.

[0058] In some cases, the interval detection algorithm includes or processes simultaneously with the event analysis algorithm running on the external computing device. The event analysis algorithm may include a deep learning algorithm that performs various layers of processing that are trained on the user's specific preferences. For example, the user may provide feedback or instructions in the event analysis algorithm to teach the event analysis algorithm about the nature and priority of the events detected by the wearable device. In one example, the event analysis algorithm is trained to identify certain speech data (e.g., a certain person, a certain category, etc.) and, upon identification, transcribe the speech data into text. The event analysis algorithm may classify events into different priority levels based on the identification, for example, a fire alarm may have a higher priority than a doorbell. In aspects, the event analysis algorithm on the computing device may access two or more microphones of the wearable device via a wireless connection, such as using short-range communication (e.g., Bluetooth). In aspects, the event analysis algorithm may be combined with or work simultaneously with the interval detection algorithm running on the wearable device. In some cases, the event analysis algorithm and the interval detection algorithm may be collectively referred to as an event determination algorithm.

[0059] FIG. 1 illustrates an example of a system 100 in which aspects of the disclosure may be practiced. As illustrated, the system 100 includes a wearable device 110 communicatively coupled to a computing device 120. The wearable device 110 is illustrated as a headset including two or more speakers and two or more microphones. The computing device 120 is illustrated as a smartphone or tablet computer wirelessly paired with the wearable device 110. At a high level, the wearable device 110 may play audio content transmitted from the computing device 120. A user may use a graphical user interface (GUI) on the computing device 120 to select audio content and / or adjust settings of the wearable device 110. The wearable device 110 may provide soundproofing, active noise cancellation, and / or other audio enhancement features for playing audio content transmitted from the computing device 120. According to aspects of the present disclosure, upon determining that an event requires the user's attention, the wearable device 110 and / or the computing device 120 can provide the user with a non-intrusive, spatialized notification or feedback regarding the event and its location attributes.

[0060] In one aspect, the wearable device 110 includes at least two microphones 111 and 112 to capture ambient sound. The captured sound may be used for active noise cancellation and / or event detection. For example, the microphones 111 and 112 are positioned on opposite sides of the wearable device 110. The audio signals captured by the microphones 111 and 112 may be correlated and / or triangulated to determine location attributes of a detected event. In aspects, additional microphones positioned at different locations of the wearable device 110 may be used to increase the accuracy of the location determination.

[0061] In one aspect, the headphones 110 include a voice activity detection (VAD) circuit capable of detecting the presence of a speech signal (e.g., a human speech signal) in a sound signal received by the microphones 111, 112 of the headphones 110. For example, the microphones 111, 112 of the headphones 110 can receive ambient external sounds near the headphones 110, including speech produced by a user. The sound signal received by the microphones 111, 112 can have the speech signal mixed with other sounds near the headphones 110. Using the VAD, the headphones 110 can detect and extract the speech signal from the received sound signal. In one aspect, the VAD circuit can be used to detect and extract the speech produced by a user to facilitate a voice call, a voice chat between the user and another person, or a voice command for a cloud-based virtual personal assistant (VPA), such as a VPA. In other examples, detection or activation may include self-VAD (which only initiates when the user is speaking, regardless of whether others in the area are speaking), active transportation (sound captured from a transportation system), head gestures, buttons, computing device-based activation (e.g., pause / unpause from a phone), changes in input audio levels, audible changes in the environment, among others. The voice activity detection circuitry may perform or assist in performing the activity detection algorithms disclosed herein.

[0062] The wearable device 110 further includes hardware and circuitry, including a processor / processing system and memory, configured to implement one or more sound management capabilities or other capabilities, including, but not limited to, noise cancellation circuitry (not shown) and / or noise masking circuitry (not shown), body movement detection devices / sensors and circuitry (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetometers, etc.), geolocation circuitry, and other sound processing circuitry.

[0063] In one aspect, the wearable device 110 is wirelessly connected to the computing device 120 using one or more wireless communication methods, including but not limited to Bluetooth, Wi-Fi, Bluetooth Low Energy (BLE), other RF-based techniques, etc. In one aspect, the wearable device 110 includes a transceiver that transmits and receives data via one or more antennas to exchange audio data and other information with the computing device 120.

[0064] In one aspect, wearable device 110 includes communications circuitry capable of transmitting and receiving audio data and other information from computing device 120. Wearable device 110 also includes an incoming audio buffer, such as a render buffer, that buffers at least a portion of the incoming audio signal (e.g., audio packets) to allow time for retransmission of any missing or missing data packets from computing device 120. For example, when wearable device 110 receives a Bluetooth transmission from computing device 120, the communications circuitry typically buffers at least a portion of the incoming audio data in a render buffer before the audio is actually rendered and output as audio to at least one of the transducers (e.g., audio speakers) of wearable device 110. This is done to ensure that even if there is an RF collision that causes an audio packet to be lost during transmission, there is time for the lost audio packet to be retransmitted by computing device 120 before it needs to be rendered by wearable device 110 for output by one or more acoustic transducers of wearable device 110.

[0065] Audio output device 110 is shown as an over-the-head headphone. However, the techniques described herein apply to other wearable devices, such as wearable audio devices, including any audio output device that fits around, on, in, or near the ear (including open-ear audio devices worn on the user's head or shoulders), or other body parts of the user, such as the head or neck. Wearable device 110 may take any form, wearable or otherwise, including a stand-alone device (including an automobile speaker system), a stationary device (including a portable device, such as a battery-powered portable speaker), a headphone, an earphone, an earpiece, a headset, goggles, a headband, an earbud, an armband, a sports headphone, a neckband, or glasses.

[0066] In one aspect, wearable device 110 is connected to computing device 120 using a wired connection, with or without a corresponding wireless connection. Computing device 120 may be a smartphone, tablet computer, laptop computer, digital camera, or other computing device that connects with wearable device 110. As shown, computing device 120 may connect to a network 130 (e.g., the Internet) and may access one or more services on the network. As shown, these services may include one or more cloud services 140.

[0067] In one aspect, computing device 120 can access cloud servers in cloud 140 on network 130 using a mobile web browser or a local software application or “app” running on computing device 120. In one aspect, the software application or “app” is a local application installed and executed locally on computing device 120. In one aspect, cloud servers accessible on cloud 140 include one or more cloud applications running on the cloud servers. The cloud applications can be accessed and executed by computing device 120. For example, the cloud applications can generate web pages that are rendered by a mobile web browser on computing device 120. In one aspect, mobile software applications installed on computing device 120 or cloud applications installed on cloud servers can be used individually or in combination to implement techniques for low latency Bluetooth communication between computing device 120 and wearable device 110 according to aspects of the present disclosure. In one aspect, examples of local software applications and cloud applications include gaming applications, audio AR applications, and / or gaming applications with audio AR capabilities. The computing device 120 can receive signals (e.g., data and control) from the wearable device 110 and send signals to the wearable device 110.

[0068] 2A illustrates an example wearable device 110 and some of its components. Other components may be unique to wearable device 110 and are not shown in FIG. 2A. For example, wearable device 110 may include a housing that houses an optional graphical interface (e.g., an OLED display) that can provide a user with information about music that is currently playing ("now playing").

[0069] The wearable device 110 includes one or more electroacoustic transducers (or speakers) 214 for outputting audio. The wearable device 110 also includes a user input interface 217. The user input interface 217 may include multiple preset indicators, which may be hardware buttons. The preset indicators may provide a user with easy one press access to entities assigned to those buttons. The assigned entities may be associated with different ones of the digital audio sources such that a single wearable device 110 may provide single press access to a variety of different digital audio sources.

[0070] The sensors 111 and 112 may include two or more microphones for capturing ambient sound and provide audio signals for determining location attributes of an event. For example, the sensors 111 and 112 may provide a mechanism for determining sound arrival differences at the wearable device 110 to ascertain location attributes. In some cases, the sensors 111 and 112 may capture transmission delays, which can be used to reduce errors in subsequent calculations. The sensors 111 and 112 may each provide two or more channels of audio signals. The audio signals are captured by microphones that are spaced apart and may have different directional responses. The two or more channels of audio signals may be used to calculate directional attributes of an event of interest.

[0071] 2A, the wearable device 110 includes an acoustic driver or speaker 214 for converting audio signals into acoustic energy through audio hardware 223. The wearable device 110 also includes a network interface 219, at least one processor 221, audio hardware 223, a power supply 225 for powering various components of the wearable device 110, and memory 227. In one aspect, the processor 221, network interface 219, audio hardware 223, power supply 225, and memory 227 are interconnected using various buses 235, and some of the components may be mounted on a common motherboard or in other manners as desired.

[0072] Network interface 219 provides communication between wearable device 110 and other electronic computing devices via one or more communication protocols. Network interface 219 provides either or both of a wireless network interface 229 and a wired interface 231 (optional). Wireless interface 229 allows wearable device 110 to communicate wirelessly with other devices according to a wireless communication protocol such as IEEE 802.11. Wired interface 231 provides network interface functionality over a wired (e.g., Ethernet) connection for reliability and fast transfer speeds, for use, for example, when wearable device 110 is not worn by a user. Although shown, wired interface 231 is optional.

[0073] In one particular aspect, the network interface 219 includes a network media processor 233 to support Apple AirPlay® and / or Apple Airplay® 2. For example, if a user connects an AirPlay® or Apple Airplay® 2-enabled device, such as an iPhone® or iPad® device, to the network, the user can then stream music to a network-connected audio playback device via Apple AirPlay® or Apple Airplay® 2. Of note, the audio playback device can support audio streaming via AirPlay®, Apple Airplay® 2, and / or DLNA® UPnP protocols, all integrated into one device.

[0074] All other digital audio received as part of a network packet can be passed straight from the network media processor 233 through a USB bridge (not shown) to the processor 221, on to a decoder, a DSP and finally rendered via the electroacoustic transducer 214.

[0075] Network interface 219 may further include Bluetooth circuitry 237 for Bluetooth applications (e.g., for wireless communication with Bluetooth-enabled audio sources such as smartphones or tablets) or other Bluetooth-enabled speaker packages. In some aspects, Bluetooth circuitry 237 may be the primary network interface 219 due to energy constraints. For example, network interface 219 may use Bluetooth circuitry 237 only for mobile applications when wearable device 110 adopts any wearable form factor. For example, BLE technology may be used in wearable device 110 to extend battery life, reduce package weight, and provide high quality performance without other backup or alternative network interfaces.

[0076] In one aspect, the network interface 219 supports communication with other devices using multiple communication protocols at the same time. For example, the wearable device 110 can support Wi-Fi / Bluetooth coexistence and can support simultaneous communication using both Wi-Fi and Bluetooth protocols at one time. For example, the wearable device 110 can receive an audio stream from a smartphone using Bluetooth and can further simultaneously rebroadcast the audio stream to one or more other devices over Wi-Fi. In one aspect, the network interface 219 can include only one RF chain capable of communicating using only one communication method (e.g., Wi-Fi or Bluetooth) at a time. In the present context, the network interface 219 can support Wi-Fi and Bluetooth communication simultaneously, for example, by time-dividing a single RF chain between Wi-Fi and Bluetooth according to a time division multiplexing (TDM) pattern.

[0077] The streamed data may be passed from the network interface 219 to the processor 221. The processor 221 may execute instructions (e.g., for performing digital signal processing, decoding, and equalization functions, among others), including instructions stored in memory 227. The processor 221 may be implemented as a chipset of chips including separate analog and digital processors. The processor 221 may provide coordination of other components of the audio wearable device 110, such as, for example, control of a user interface.

[0078] In certain aspects, the protocols stored in memory 227 may include, for example, BLE according to Bluetooth Core Specification version 5.2 (BT5.2). Wearable device 110 and various components therein are provided herein to fully conform to or implement aspects of the protocols and associated specifications. For example, BT5.2 includes an enhanced attribute protocol (EATT) that supports concurrent transactions. To support EATT, a new L2CAP mode is defined. Thus, wearable device 110 includes sufficient hardware and software components to support the BT5.2 specifications and modes of operation, even if not explicitly shown or discussed in this disclosure. For example, wearable device 110 may utilize LE isochronous channels specified in BT5.2.

[0079] Processor 221 provides the processed digital audio signals to audio hardware 223, which includes one or more digital-to-analog (D / A) converters for converting the digital audio signals to analog audio signals. Audio hardware 223 also includes one or more amplifiers that provide amplified analog audio signals to electroacoustic transducer 214 for sound output. In addition, audio hardware 223 may include circuitry for processing analog input signals to provide digital audio signals for sharing with other devices, for example, other speaker packages for synchronous output of digital audio.

[0080] The memory 227 may include, for example, flash memory and / or non-volatile random access memory (NVRAM). In some aspects, the instructions (e.g., software) are stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., processor 221), perform one or more processes, such as those described elsewhere herein. The instructions may also be stored by one or more storage devices, such as one or more computer-readable or machine-readable media (e.g., memory 227, or memory on a processor). The instructions may include instructions for performing decoding (i.e., a software module includes an audio codec for decoding a digital audio stream), as well as digital signal processing and equalization. In certain aspects, the memory 227 and processor 221 may cooperate with the feedback microphone 111 and the feedforward microphone 112 in data acquisition and real-time processing.

[0081] FIG. 2B illustrates an exemplary computing device 120, such as a smartphone or mobile computing device, according to certain aspects of the present disclosure. Some components of the computing device 120 may be unique and are not illustrated in FIG. 2B. For example, the computing device 120 may include a housing. The housing may house an optional graphical interface 212 (e.g., an OLED display) as shown. The graphical interface 212 provides information to the user about the currently playing music or video ("now playing"). The computing device 120 includes one or more electroacoustic transducers 215 for outputting audio. The computing device 120 may also include a user input interface 216 to enable user input.

[0082] Computing device 120 also includes a network interface 220, at least one processor 222, audio hardware 224, a power supply 226 for powering various components of computing device 120, and memory 228. In an embodiment, processor 222, graphical interface 212, network interface 220, audio hardware 224, power supply 226, and memory 228 are interconnected using various buses 236, and some of the components may be mounted on a common motherboard or in other manners as needed. In some embodiments, processor 222 of computing device 120 is more powerful in terms of computational capabilities than processor 221 of wearable device 110. Such differences may be due to weight, power supply, and other requirement constraints. Similarly, power supply 226 of computing device 120 may be of larger capacity and heavier weight than power supply 225 of wearable device 110.

[0083] Network interface 220 provides communication between computing device 120 and wearable device 110, as well as between other audio sources and other wireless speaker packages, including one or more networked wireless speaker packages and other audio playback devices, via one or more communication protocols. Network interface 220 may provide either or both of a wireless interface 230 and a wired interface 232 (optional). Wireless interface 230 enables computing device 120 to communicate wirelessly with other devices according to a wireless communication protocol such as IEEE 802.11. Wired interface 232 provides network interface functionality via a wired (e.g., Ethernet) connection.

[0084] In certain aspects, network interface 220 may also include network media processor 234 and Bluetooth circuitry 238, similar to network media processor 233 and Bluetooth circuitry 237 in wearable device 110 in Figure 2A. Additionally, in aspects, network interface 220 supports communication with other devices using multiple communication protocols at once, as described with respect to network interface 219 in Figure 2A.

[0085] All other digital audio received as part of a network packet comes straight from the network media processor 234 via the USB bridge 236 to the processor 222, then to the decoder, the DSP, and finally to be rendered via the electroacoustic transducer 215.

[0086] The computing device 120 may also include an image or video acquisition unit 280 for capturing image or video data. For example, the image or video acquisition unit 280 may be connected to one or more cameras 282 and capture still or moving images. The image or video acquisition unit 280 may operate at various resolutions or frame rates according to user selection. For example, the image or video acquisition unit 280 may capture 4K video (e.g., resolution of 3840×2160 pixels) by one or more cameras 282 at 30 frames per second, FHD video (e.g., resolution of 1920×1080 pixels) at 60 frames per second, or slow-motion video at lower resolutions, depending on the hardware capabilities of the one or more cameras 282 and user input. The one or more cameras 282 may include two or more individual camera units with respective lenses of different characteristics, such as focal lengths, that provide different fields of view. The image or video acquisition unit 280 may switch between two or more individual camera units of the cameras 282 during continuous recording.

[0087] The captured audio or audio recording, such as a voice recording captured at the wearable device 110, may be passed from the network interface 220 to the processor 222. The processor 222 executes instructions in the wireless speaker package, including instructions stored in memory 228 (e.g., for performing digital signal processing, decoding, and equalization functions, among others). The processor 222 may be implemented as a chipset of chips including separate analog and digital processors. The processor 222 may provide coordination of other components of the audio computing device 120, such as, for example, control of user interfaces and applications. The processor 222 provides processed digital audio signals to the audio hardware 224, similar to the respective operations by the processor 221 described in FIG. 2A.

[0088] The memory 228 may include, for example, flash memory and / or non-volatile random access memory (NVRAM). In an aspect, the instructions (e.g., software) are stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., processor 222), perform one or more processes, such as those described herein. The instructions may also be stored by one or more storage devices, such as one or more computer-readable or machine-readable media (e.g., memory 228, or memory on processor 222). The instructions may include instructions for performing decoding (i.e., the software module includes an audio codec for decoding a digital audio stream), as well as digital signal processing and equalization.

[0089] Examples of localized feedback hearing enhancement and wearable systems Aspects of the present disclosure provide techniques, including devices and systems implementing the techniques, for providing feedback of events to a user while the user is wearing a wearable device. For example, the wearable device can provide a user with high quality noise-canceled audio playback, reducing the user's situational awareness. The feedback can inform the user about events that merit attention through spatialized audio cues, with minimal intrusion into the noise-canceled playback experience. The techniques include measuring ambient sound using two or more microphones on the wearable device. In some cases, at least three microphones are positioned to measure ambient sound surrounding the user. An event is determined based on the measured ambient sound, and location attributes of the event are determined. Deep learning algorithms can be used to identify the nature and / or classification of the event and filter out events that the user prefers to ignore. Upon determining an event that requires the user's attention, the user is provided with feedback of the determined event and location attributes, the feedback indicating the nature and location of the event.

[0090] In an aspect, the wearable device may play audio content through two or more speakers thereon that cancels the measured ambient sound. For example, the two or more speakers may play audio content that has the same amplitude as the ambient sound and an inverted phase relative to the ambient sound. Thus, interference caused by the noise canceling audio content cancels the ambient sound. In an aspect, providing feedback to the user based on the event and location attributes includes playing spatialized audio cues in addition to the noise canceling audio content. In this way, the user can maintain situational awareness while escaping ambient sound.

[0091] As an example, determining an event based on the measured ambient sound may include measuring the loudness of the sound level of the event and using an audio signal associated with the event to determine the location of the event relative to the wearable device. The location of the event may be determined based on the direction and distance processed from the audio signal. The location attribute of the event may include at least one of the direction or distance. For example, a user may not want to be notified about the user's own speech. Events associated with a distance within a certain threshold may be determined as the user's own speech and thus ignored. Similarly, a user may be immersed in a noise cancellation experience but still be able to recognize events that are visually identifiable. For example, if the user's colleague is speaking in front of the user, the user does not need to listen to the speech to recognize the speech because the colleague's lip movements can provide sufficient visual notification to the user. Thus, the location attribute is an important aspect of determining events that may or may not require the user's attention.

[0092] FIG. 3 illustrates an example of operations 300 for providing a user with feedback of determined events and location attributes according to certain aspects of the disclosure. The operations 300 begin at 302 by measuring ambient sound using two or more microphones. In some cases, at least three microphones are positioned to measure ambient sound surrounding the user. For example, block 302 may be implemented by the wearable device 110 and microphones 111 and 112 as shown in FIG. 1. An additional example is shown in FIG. 4, where microphones 413, 415, and 425 are used to measure ambient sound. The measured ambient sound may be used to generate noise canceling content and monitored for events requiring the user's attention, as described further below.

[0093] At 304, the event determination algorithm may determine an event based on the measured ambient sound. For example, the event determination algorithm may measure the loudness of the sound level of the event and use an audio signal associated with the event to determine the location of the event relative to the wearable device. The event determination algorithm may determine that the loudness of the sound level of the event exceeds a threshold that corresponds to the location of the event relative to the wearable device (e.g., different relative locations may have different thresholds). For example, the threshold may vary based on the location of the event relative to the wearable device. The threshold may increase as the location of the event gets closer to the user, such that sounds produced by the user may be eliminated.

[0094] In some cases, the event determination algorithm may monitor and calculate the average amplitude (i.e., decibel level) of the background noise and consider incoming sounds that exceed the average decibel level as event candidates. In some cases, the event determination algorithm may use a user-input threshold for decibel level in determining event candidates. In some cases, the event determination algorithm may use ambient sound history or patterns as criteria for identifying abnormally loud sounds. A combination of two or more of the example strategies may be implemented.

[0095] At 306, the event determination algorithm determines location attributes of the event. The location attributes of the event may include at least one of a direction relative to the user / wearable device and a distance between the event and the user / wearable device. In some cases, the location of the event may be determined based on the direction and distance processed from the audio signal. For example, two or more microphones allow the event determination algorithm to correlate and / or triangulate sounds of similar characteristics to determine the location. For example, due to different travel distances, the same sound arrives at each of two or more microphones at different times, allowing the determination of the relative location for the two or more microphones. Thus, the direction and distance can be processed from the audio signal to determine the location of the event. As shown in FIG. 4, different amounts of delay of the same sound signal (e.g., measurements plotted in graphs 410 and 420 for pairs 413 and 415, and 415 and 425, respectively) allows the event determination algorithm to perform far-field localization of the sound signal, as shown in subfigure 430.

[0096] In some cases, the event determination algorithm may determine an event based on ambient sound by filtering out incidents of the event based on the location of the event relative to the wearable device. For example, the event determination algorithm may filter out incidents that are within a certain distance to the wearable device (e.g., classifying incidents caused by the user, such as the user's own speech), that are within the field of view or direction (such as in front) of the wearable device (e.g., classifying incidents that are visually noticeable by the user, such as a computer notification sound from a speaker in front of the user, someone talking in front of the user), or that may be too far away to require the user's attention (e.g., incidents of traffic noise outside the residence). Some examples are further illustrated in FIG. 5 and discussed below.

[0097] In some cases, the event determination algorithm may determine an event based on ambient sounds by filtering out incidents of an event based on recurring patterns in the audio signal associated with the event. For example, the event determination algorithm may analyze a history of recorded ambient sounds to identify patterns of ambient sounds that were ignored by the user and determine whether a detected event fits this pattern and should be ignored.

[0098] In some cases, an event determination algorithm may determine an event based on ambient sounds by filtering out incidents of an event based on attributes of the audio signal associated with the event. For example, attributes of the audio signal may indicate where, what, or who the audio signal is associated with. An event determination algorithm may distinguish an inside fire alarm from an outside siren, or distinguish speech from an acquaintance (e.g., based on previously recorded events) from speech from a stranger.

[0099] In some cases, the event determination algorithm may determine an event based on correlating the location of the event relative to the wearable device, repetitive patterns in the audio signal associated with the event, and attributes of the audio signal associated with the event. For example, the event determination algorithm may use deep learning to identify the nature or meaning of the audio signal of the event. Deep learning, or machine learning in a more general context, may use artificial neural networks with representation / feature learning performed by artificial intelligence. As discussed further below with respect to Figures 6-7, the event determination algorithm may use artificial intelligence to perform deep learning on data of the location of the event relative to the wearable device, repetitive patterns in the audio signal associated with the event, and attributes of the audio signal associated with the event. Deep learning may accurately determine which events merit the user's attention and ignore false positive events.

[0100] At 308, feedback of the determined event and location attributes is provided to the user. For example, the feedback may include playing spatialized sound through two or more speakers in the wearable device, providing a directional haptic notification using the wearable device, displaying a notification on one or more devices connected to the wearable device, or a combination thereof. The spatialized sound may be produced using two or more speakers of the wearable device (e.g., speakers capable of producing surround sound). The spatialized sound may represent the location (e.g., location attributes) of the determined event relative to the user. The wearable device may include two or more haptic feedback mechanisms for providing directional input (e.g., left / right, up / down, etc.). FIG. 5 illustrates an example of displaying a notification in the form of a visual presentation 500 of a recorded incident.

[0101] As shown in FIG. 5, the visual representation 500 plots detected incidents over left-to-right (LR) delays (i.e., time delays indicating relative distance to the wearable device) on the x-axis and over front-to-back (FB) delays on the y-axis. A circular range 540 indicates the range of delay pairs that are consistent with far-field sound sources. The visual representation 500 includes a user's field of view 520 and an associated forward direction 530. Incidents detected within the user's field of view 520 may be assumed to be negligible. The forward direction 530, which is aligned with the wearable device and undergoes changes (e.g., as the user moves), may be used to update the coverage of incidents within the field of view 520. For example, the forward direction 530 may be defined when the wearable device is initialized and may be updated using input from one or more motion sensors, such as an accelerometer, a gyroscope, or an inertial measurement unit (IMU).

[0102] A number of examples of incidents 510, 512, and 514 are shown in FIG. 5. Incident 510 represents an event that exceeds a threshold sound level and falls within the field of view 520. As a result, incident 510 is not reported to the user. Incident 512 represents an event that exceeds a threshold sound level and falls outside the field of view 520. Furthermore, incident 512 is associated with a state of non-concentration of the user and is not reported to the user. Incident 514 represents an event that exceeds a threshold sound level, falls outside the field of view 520, and is associated with a state of concentration of the user. Incident 514 is reported to the user along with associated location attributes, which may be presented as spatialized audio cues and by visual representation 500.

[0103] For example, the event determination algorithm may determine that the user is in a concentration state based on a pattern of motion signals measured by the wearable device. The pattern of motion signals may be measured by one or more IMUs located on the wearable device. As the user's situational awareness decreases in a concentration state, such as when the user is sitting still and reading or watching, feedback regarding a detected incident is more valuable in such a concentration state. An example 800 of a measurement of a motion signal is shown in FIG.

[0104] With brief reference to FIG. 8, the x-axis 802 represents time and the y-axis 804 represents energy, typically measured by a gyroscope or motion sensor. As shown, a low energy level pattern 816 represents a user's concentration state, while a high energy level pattern 818 represents a moving state. When the user is not in a concentration state, the incident 512 is ignored because the sound signal may include noise generated due to the user's own movement. In some cases, determining that the user is in a concentration state may include processing the pattern of the motion signal. The pattern of the motion signal need not indicate that the user is sitting or standing still, but rather indicates a repeating pattern, such as pacing or rocking back and forth.

[0105] One important aspect of the present disclosure relates to determining an event (such as incident 514) using deep learning based on various layers of data, including audio attributes, location attributes, and user status. At a high level, the deep learning aspect detects events based on multiple classes or categories by analyzing various attributes of the audio signal (e.g., distinguishing the source and nature of the sound signal). Second, the deep learning aspect can determine the class or category of each detected event (e.g., voice, alarm, knock, etc.) and associated priority level. Deep learning can be supervised (e.g., always receiving user feedback), semi-supervised (e.g., occasionally receiving user feedback), and unsupervised (e.g., not receiving user feedback).

[0106] FIG. 6A illustrates an example of a process for determining an event requiring user attention, according to certain aspects of the present disclosure. As shown, two or more feature extraction modules may receive an audio time series in one or more channels. Features are then extracted and provided to a class-wise energy prediction network (e.g., a machine learning network or a neural network) and a sound localization processing network to determine location attributes of the extracted features. In some cases, the feature extraction is a deterministic process. In some cases, the feature extraction produces a "mel spectrogram" image. For audio time series in multiple channels, the feature extraction process can be followed by processing in a sound localization network that processes the directional output of the ambient sound.

[0107] The two networks can then determine what feedback may be provided to the user by applying an event threshold and decision logic based on the event threshold and decision logic module, which then outputs a decision per sound class. In some cases, the example of FIG. 6A uses thresholding the rms estimate for each sound class (e.g., not limited to 0-1) to determine the corresponding detection sequence. In some cases, the detection thresholding process can apply a per-class threshold or a custom threshold to the event detection. The output signal in FIG. 6A ("Class N Decision") can define what the user is notified of with feedback.

[0108] A second example of a deep learning process 600 for determining an event requiring user attention is shown in FIG. 6B. The deep learning process 600 may be implemented as an alternative (e.g., separate from) the example shown in FIG. 6A, or may be implemented together with the example shown in FIG. 6A, as in another example shown in FIG. 6C below. As shown, the deep learning process 600 begins at 610 by receiving input audio. Although shown as a single channel audio stream, the input audio often includes at least two recorded channels (e.g., for localization analysis). Two or more channels of the input audio may be analyzed in parallel. At 620, features in the input audio are extracted. Features may be identified based on magnitude, frequency, pattern, or a combination thereof.

[0109] At 630, a filter is applied to the extracted features to further classify events associated with the input audio features. For example, the feature extraction process may include filtering and transformation on the input audio signal (e.g., converting to a frequency domain "mel spectrogram"). In some cases, a "per-class" energy prediction network implementing the process at 630 may include a neural network that predicts the rms energy per class based on the extracted features. In some cases, the feature extraction process in FIG. 6B may differ from the process performed in "Feature Extraction" of FIG. 6A.

[0110] At 640, a localization analysis based on two or more channels of the input audio is performed, resulting in two identification outputs 650 and 654. Output 650 includes sound event detection with multi-label classification that may indicate the nature of the detected event, such as speech 641, a sound associated with a car horn 642, a dog barking 643, or a siren (not shown). The detected sounds are recorded along a timeline T, with each frame having a duration of t. Output 654 provides a direction of arrival estimate, such that the direction of the detected events, such as speech 641, a sound associated with a car horn 642, a dog barking 643, and the like, may be determined by determining the XYZ coordinates of the origin of each event.

[0111] In some cases, the "sigmoid" layer leading to 650 can implement a probability-based detection approach, where the probability between 0 and 1 (which is what the sigmoid block produces) is thresholded to determine the detection sequence for each class as in 650. In some cases, the process at 630 can be implemented using a non-recursive network. The process at 640 can be implemented using a recursive network.

[0112] 6C shows another example of using a neural network to determine if and how feedback can be provided to a user. As shown, FIG. 6C shows thresholding and decision steps that are separate.

[0113] FIG. 7 shows the output of an exemplary deep learning model. The output is an energy time series prediction per class. A detection decision can be produced by applying a threshold to the prediction. As shown in FIG. 7, plot 705 shows using sound energy (y-axis) to determine measured and predicted events along time (frames in x-axis). For example, three example energy profiles 710, 712, and 714 are shown. Profile 710 may include speech and other sound signals, which may correspond to audio signal 723 in graph 725 on the right. Profile 712 represents the amount of energy of the speech signal that is present. An event determination algorithm can be trained to predict the expected energy fluctuations of speech-only activity and ignore the energy of non-speech events, as shown in profile 714. For example, in graph 725, events 731 or 721 represent the speech-only energy portion of the recorded signal 723. Signal 722 shows high energy during knock sound 732, while the speech energy portion during this time remains small.

[0114] In some aspects, the techniques variously described herein may be used to determine contextual information about a wearable device and / or a user of the wearable device. For example, the techniques may be used to help determine aspects of the user's environment (e.g., noisy, quiet, indoors, outdoors, on an airplane, in a car, etc.) and / or activity (e.g., commuting, walking, running, sitting, driving, flying, etc.). In some such aspects, audio data received from the wearable device may be processed at a computing device to determine such contextual information and provide a new or enhanced experience to the user. For example, this may enable customization of playlists or audio content, noise cancellation adjustments, and / or other setting adjustments (e.g., audio equalizer settings, volume settings, notification settings, etc.), to name a few.

[0115] Because wearable devices (e.g., headphones or earphones) typically have limited resources (e.g., memory and / or processing resources), using the techniques described herein to offload processing of data from sensors of a wearable device to a computing device while having a system for synchronizing audio data at the computing device provides a variety of applications. In some aspects, the techniques disclosed herein enable a computing device to automatically identify an optimized or most preferred configuration or settings for synchronized audio capture operations, as discussed below.

[0116] Example of feedback based on customized configuration of different thresholds Aspects of the present disclosure further provide techniques, including devices and systems implementing the techniques, for providing a user with feedback of events defined by customized thresholds that vary for different constituent events corresponding to different situational criteria. For example, a wearable device may use different thresholds to determine an event in different situations. The different thresholds may be configured by a user, set by default, updated by a service, or updated based on new data (e.g., machine learning). In aspects, the thresholds are configurable decibels or volume levels that correspond to situations characterized as constituent events. Each constituent event may be defined by one or more parameters, such as a sound class (e.g., speech, melody, alarm, bark, etc.), location attributes (e.g., distance and direction from the wearable device, as described above), time, date, location of the wearable device (e.g., home, office, etc.), and other parameters. For example, a user may set different thresholds for the same sound class in different rooms of the home, such that a particular sound (e.g., a doorbell) that is attenuated may still trigger feedback to the user. The parameters are generally referred to as situational criteria to include various types of data.

[0117] In an aspect, each of the two or more different situation criteria may include customization of at least one of a threshold associated with the determined event, a location attribute of the event, or feedback to the user. In some cases, the situation criteria further includes a location attribute for the wearable device determined based on ambient sounds measured using two or more microphones of the wearable device. For example, a user-configurable threshold may include a minimum or maximum distance corresponding to one particular situation criterion. The threshold may also include a direction corresponding to one particular situation criterion. The situation criteria may correspond to a particular physical location of the wearable device. For example, a situation criterion of sound distance and direction for a home location will be different from a situation criterion of sound distance and direction for an office location. A user may configure the situation criteria such that when at home, a certain sound from a first distance in a first direction and exceeding a first threshold may result in feedback being provided to the user, while in the office, a certain sound from a second distance in a second direction and exceeding a second threshold may result in another feedback.

[0118] 9 illustrates examples of operations 900 for providing a user with feedback of a determined event according to certain aspects of the disclosure. The operations 900 may be performed at least in part by a wearable device, such as the wearable device 110 of FIG. 1. The operations 900 differ from the operations 300 of FIG. 3 in that the operations 900 may determine the event by applying situation-specific thresholds that vary in different situations characterized by different parameters or situation criteria. For example, in addition to using location attributes as discussed in the operations 300, the operations 900 also consider customized sensitivity or thresholds, sound characteristics (e.g., sound type or class), as well as customized feedback, as discussed below.

[0119] Operation 900 begins at 902 by measuring ambient sound using two or more microphones. This is similar to operation 300. Two or more microphones allow detecting the ambient sound position relative to a wearable device such as wearable device 110 and microphones 111 and 112 as shown in FIG. 1. For example, multiple microphones may allow estimation of the direction of arrival by estimating the time delay between different pairs of microphones. Distance may be indirectly determined from sound intensity, reverberation, and some other quantities. Other location determination techniques may be applied as well. An additional example is shown in FIG. 4, where microphones 415 and 425 are used to measure ambient sound. The measured ambient sound may be used to generate noise cancellation content and monitored for events requiring the user's attention.

[0120] At 904, an event may be determined based on the measured ambient sound exceeding a threshold. The threshold may vary for two or more different constituent events corresponding to two or more different situation criteria. For example, the threshold may be customized by a user or vary by default depending on different sound characteristics. In this way, the wearable device may behave differently in different situations, such as during a call in a conference setting, in an audio playback scenario, or in a distraction mode that eliminates all surrounding environmental noise.

[0121] At 906, corresponding feedback is provided to the user based on the determined event. For example, the feedback may include playing spatialized sound via two or more speakers in the wearable device, providing directional haptic notifications using the wearable device, displaying notifications on one or more devices connected to the wearable device, or combinations thereof, as discussed above.

[0122] In an aspect, each of the two or more different situation criteria includes a customization configuration of at least one of a threshold associated with the determined event, a location attribute of the event relative to the wearable device, or feedback to the user based on the determined event. The customization configuration may be input by a user. The customization configuration may include a default factory configuration. For example, FIGS. 10 and 11 show examples of interfaces for adjusting the customization configuration of the situation criteria according to certain aspects of the present disclosure.

[0123] As shown in FIG. 10, a user can manage notifications by selecting one or more of the notification types, such as audio alerts, desktop notifications, or both. In addition to notification types, the customization configuration shown in FIG. 10 can also include event sensitivity definitions for different sound classes, such as speech, alarm / timer, and loud events as shown. Event sensitivity can dictate the minimum magnitude (e.g., valley) of a sound wave or the range of frequencies that can be detected. In some cases, the sensitivity can be descriptive and intuitive to the user, such as by providing feedback for "speech in the same room" and "speech in the next room." Part of the algorithm can match the measured sound levels to these descriptions depending on the particular scenario. The selected event sensitivity can be associated with a decibel level that defines an event threshold. The threshold dictates the minimum sound level that must be exceeded to trigger feedback to the user. In other words, the wearable device or a processing device connected with the wearable device can first identify the appropriate sound class of ambient sound and monitor when such ambient sound exceeds a defined threshold.

[0124] In some cases, the sound classes may include at least one of speech, background sounds (e.g., loud events shown in FIG. 10), or musical sequences (i.e., variations in sounds having a discernible pattern). In some examples, background sounds may also include alarms, footsteps, traffic noise, or general sound patterns. In some cases, FIG. 10 may represent a graphical user interface (GUI). The GUI may be on the wearable device, on a computing device connected to the wearable device, or both. The computing device may be any computing device that communicates with the wearable device, such as computing device 120 of FIG. 1.

[0125] FIG. 10 shows that the event sensitivity settings for the example sound classes include discrete optional levels: off, low, medium, and high (or corresponding user-friendly descriptions), although a continuous sensitivity level configuration, such as in the form of a slide bar, may be provided. In some cases, complex threshold profiles (such as magnitude profiles over a range of frequencies) may be loaded into the configuration. Furthermore, in some cases, threshold configurations may be trained, acquired, or identified by the wearable device (e.g., by machine learning) based on the user's response to ambient sounds. For example, the user's response may be measured using various motion sensors in addition to audio responses.

[0126] Sound class selection, detection sensitivity, and feedback or notification mode may be configured by a user, as shown in Fig. 11. Fig. 11 may be considered an example of a graphical interface implementing various aspects of Fig. 10.

[0127] In an aspect, the feedback provided to the user further includes playing a recorded copy of the ambient sound to the user. Upon determining an event that meets certain situation criteria, such as, for example, a speech from a coworker in a work setting, the ambient sound (i.e., the speech) is recorded and played to the user at the wearable device, which may initially (i.e., before processing the environmental sounds and identifying that the ambient sound meets the speech situation criteria) cancel the speech. The recording of the ambient sound includes at least two channels (e.g., binaural) to preserve the spatial characteristics of the ambient sound and provide location attributes as discussed in various aspects herein. Playing the recorded sound copy may be referred to as one or more operations (e.g., automatic ambient sound capture, event replay, etc.) described below. In this way, even though the wearable device may initially cancel the ambient noise completely, important event or audio information is preserved and available to the user.

[0128] In aspects, notifications or feedback may be based on: 1: spatialized tones or cues; 2: automatic ambient sound capture (i.e., automatically changing the headphones to an ambient sound capture noise cancellation state); and 3: event replay, also referred to as "pass-through." In some cases, playing back recorded events may have some delay relative to what the user hears in real life, as the processor may need time to make a detection decision or process / enhance the sound event. In some cases, a built-in "rewind" may be performed when the noise cancellation state is not changed.

[0129] For example, automatic ambient sound capture can put the wearable device into a mode where the device passes outside world audio in real time in a manner that attempts to best simulate what it prefers. Event replay can record a sound event and then play that sound event at some time in the future, such as in response to the user providing input that they would like to hear the event (e.g., shaking their head, pressing a button, etc.).

[0130] In another example, automatic ambient sound capture may be triggered, for example, when the general sound around the user exceeds a certain decibel level, allowing the user to know what is going on in the area, etc. However, event replay may be triggered when speech is detected within a predefined window based, for example, on the opening of a user's cubicle, where the speech is recorded and then played back to the user (which may cause the wearable to go into automatic ambient sound capture mode, allowing the user to participate in the conversation).

[0131] In aspects, one's own voice (e.g., detected using known voice activity detection (VAD) techniques) may be used to keep the wearable in a capture state after the algorithms of the present application have caused the headphones to enter a capture state (using automatic capture). The own voice activation option feature may be set by default or user customizable.

[0132] In aspects, a timer (which may be set to a default time and / or user customizable) may optionally be used to determine when to exit the automatic ambient sound capture mode (e.g., return to the noise cancellation level set before automatic ambient sound capture was activated). Alternatively, exiting the automatic ambient sound capture mode may occur in other manners, such as in response to user input (e.g., button press, head rotation).

[0133] In aspects, when the user can clearly hear speech regardless of the presence of the wearable device (e.g., based on sound levels detected by the feedback microphone), the wearable device can automatically play audio and change state to capture without requiring the user to perform further actions on the wearable device or remove the wearable device. In other words, the wearable device automatically enters capture mode when outside speech is detected.

[0134] In some cases, the wearable device may use an internal microphone (e.g., a feedback microphone near the wearable device's speaker) to determine whether a detected event is initially audible to the user, for example, in cases where the wearable device does not completely cancel the ambient sound. If the ambient sound is indeed audible to the user, the wearable device may not immediately play back the recorded event, but may provide the user with a notification of the availability of the recording. In some cases, the wearable device may amplify or attenuate the recorded copy of the ambient sound based on the data collected by the internal microphone. For example, the wearable device may adjust the volume of the playback to be substantially similar to the existing audio output so that the user can comfortably listen to the recording. In some cases, the amplification or attenuation of the recorded copy of the ambient sound may be applied to a certain specific bandwidth or content of the ambient sound. For example, the background noise of the ambient sound may be attenuated while speech may be amplified, or vice versa. Other digital processing may also be used to extract data of interest to the user.

[0135] In some cases, amplifying or attenuating the recorded copy of the ambient sound can be triggered by movement of the wearable device. For example, detection or measurement of the wearable device turning or moving toward a sound source can trigger actions to modify the recorded copy, including replaying the recorded copy (immediately or with a preconfigured delay in response to the movement), as well as amplifying or attenuating at least a portion of the recorded copy of the ambient sound. Movement data can be combined with other response data collected from the user to analyze response patterns (or movement or movement patterns), as discussed below.

[0136] In some cases, the internal microphone may be used to provide feedback control over the recording playback so that amplification or attenuation may conform to certain user preferences or profiles. For example, when the ambient sound includes a mix of sounds of different sound classes, such as speech and a fire alarm over a background song, the internal microphone may determine that the user has already heard the fire alarm (i.e., the wearable device has not completely muted the alarm sound), and playback of the recorded ambient sound need not include the fire alarm component. Similarly, the speech component may only have been picked up by the microphone of the wearable device, and the internal microphone may detect that the user was unable to perceive the speech component over other sounds, and as a result, the wearable device may amplify the speech component on a larger scale than the background song.

[0137] In some cases, the playback of the recorded ambient noise may be delayed, postponed, or offset to another time. This may be referred to as a rewind operation. A rewind operation may be implemented when the recorded ambient sound contains complex signals, when more processing time is needed, or when a user customizes or provides certain rules so that sounds of certain sound classes or certain location attributes can trigger such a rewind operation.

[0138] In some cases, the rewind and other sound replay operations (e.g., including amplification and attenuation) may be correlated with the user's motion patterns. For example, as discussed above, general rules of configuration may be identified based on input by the user. The input may be motion responses recorded by one or more motion sensors in the wearable device, such as an accelerometer, an inertial measurement unit (IMU), a vibration motion sensor, or other motion sensor. Movement patterns, such as repetitive head rotation responses, walking responses, or other movement responses, may be correlated with sound events and rewind or replay operations. In this way, when both sound events and motion responses are detected, the wearable device may automatically perform the rewind or replay operation without further input from the user.

[0139] In an aspect, the wearable device can correlate the ambient sound with location attributes of the ambient sound over a period of time, i.e., track the movement of the ambient sound. The tracked movement can be used to generate a saliency map, which maps the variation of the ambient sound over a period of time at different locations. The wearable device can also measure its own movement record over the same period of time. Based on the saliency map and the movement record of the wearable device, the user's response can be determined based on the correlation. The user's response can be conditional on the movement of the ambient sound, such as, for example, an approaching speech can induce a user's tendency toward the sound direction, and then a rewind or replay action can be performed.

[0140] In aspects, due to concerns over power consumption or processing power, the wearable device may request other connected devices to share the data processing workload. For example, a multi-stage processing architecture may be implemented to conserve bandwidth or processing power in the wearable device. In one example, the wearable device may transmit (partially or completely) the measured ambient sound to a second device, such as the computing device 120 of FIG. 1, for partial or complete calculation or processing. In some cases, the second or additional device may have greater processing power than the wearable device to reduce the overall processing time. After processing, the wearable device may receive and provide the determined events and corresponding feedback to the user.

[0141] In some aspects, the techniques variously described herein may be used for a number of audio / video applications. In addition, the techniques described herein may enable wirelessly captured audio for audio or video messages without interrupting the user's music or audio playback. Thus, the techniques described herein enable the ability to create immersive and / or noiseless audio for video using a wireless configuration. Furthermore, as can be appreciated based on this disclosure, the techniques described enable schemes that were previously only achievable using a wired configuration, and thus, the techniques described free users from the undesirable and uncomfortable experience of being tethered by one or more wires.

[0142] Although the description of the aspects of the present disclosure has been presented above for illustrative purposes, it may be noted that the aspects of the present disclosure are not intended to be limited to any of the disclosed aspects. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described aspects.

[0143] Above, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the specific described embodiments. Aspects of the present disclosure may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware embodiments, which may all be generally referred to herein as "components," "circuits," "modules," or "systems." Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.

[0144] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer readable storage media include an electrical connection having one or more wires, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present context, the computer readable storage medium may be any tangible medium that can contain or store a program.

[0145] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects. In this regard, each block of the flowcharts or block diagrams may correspond to a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or in some cases, the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, as well as combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented in a dedicated hardware-based system that performs a specific function or operates a combination of dedicated hardware and computer instructions.

Claims

1. 1. A method for providing feedback to a user of a wearable device, comprising: Measuring ambient sound using two or more microphones on the wearable device; determining an event based on the measured ambient sound; and determining a location attribute of the event relative to the wearable device; providing feedback to the user based on the event and the location attributes.

2. The method of claim 1 , wherein providing feedback to the user based on the event and the location attributes comprises playing spatialized audio cues in addition to audio content that cancels the ambient sound.

3. determining the event based on the measured ambient sound; measuring the loudness of the sound level of the event; and and determining a location of the event relative to the wearable device using an audio signal associated with the event, wherein the location of the event is determined based on a direction and distance processed from the audio signal, and the location attributes of the event include at least one of the direction or the distance.

4. determining the event based on the ambient sound; the location of the event relative to the wearable device; a repetitive pattern in the audio signal associated with the event; or The method of claim 3 , further comprising filtering out incidents of an event based on at least one attribute of the audio signal associated with the event.

5. Determining the event based on the measured ambient sound includes: The method of claim 3 , comprising determining that the magnitude of the sound level of the event exceeds a threshold corresponding to the location of the event relative to the wearable device.

6. The method of claim 5 , wherein the threshold varies based on the location of the event relative to the wearable device, the threshold increasing as the location of the event gets closer to the user.

7. 5. The method of claim 4, wherein determining the event based on the ambient sound is based at least in part on correlating i) the location of the event relative to the wearable device, ii) the repetitive patterns in the audio signal associated with the event, and iii) the attributes of the audio signal associated with the event.

8. 8. The method of claim 7, wherein determining the event is based at least in part on deep learning analysis performed by artificial intelligence using data of the location of the event relative to the wearable device, the recurring patterns in the audio signal associated with the event, and the attributes of the audio signal associated with the event.

9. Providing feedback to the user based on the event; playing spatialized sound using at least one speaker of the wearable device; providing directional tactile notifications using the wearable device; or The method of claim 1 , comprising one or more of: displaying a notification on one or more devices connected to the wearable device.

10. The method of claim 9 , wherein the spatialized sound represents a location of the determined event relative to the user, the represented location being based on at least the location attribute.

11. 10. The method of claim 1, further comprising: determining that the user is in a state of concentration based on a pattern of motion signals measured by the wearable device; and providing feedback to the user based on the event and the location attributes based on the determined state of concentration.

12. 12. The method of claim 11, wherein determining that the user is in the focused state includes processing patterns of the motion signals measured by one or more inertial measurement units (IMUs) disposed on the wearable device.

13. The method of claim 1 , wherein the two or more microphones on the wearable device include at least three microphones positioned to measure the ambient sounds surrounding the user.

14. The method of claim 1 , wherein the wearable device is an audio device worn on at least one of the user's head or neck.

15. 10. The method of claim 1, wherein the wearable device comprises at least one of over-ear headphones, on-ear headphones, in-ear headphones, earphones, completely wireless earphones, glasses, a headband, a neckband, an ear hook, an ear clip, a head-worn audio device, or a neck-worn audio device.

16. The method of claim 1 , wherein the wearable device is a noise-canceling audio device.

17. 1. A system comprising: A wearable device having two or more microphones configured to measure ambient sound; a computing device connected to the wearable device, the computing device comprising: determining an event based on the measured ambient sound; determining a location attribute of the event relative to the wearable device; and The system is configured to provide feedback to a user based on the event and the location attributes.

18. The wearable device further comprises two or more speakers, the two or more speakers being: playing audio content that cancels the measured ambient sound; and 20. The system of claim 17, configured to alert the user to the determined event by playing a spatialized audio cue in addition to the audio content.

19. 20. The system of claim 17, wherein the wearable device further comprises an actuator for providing haptic feedback in response to the computing device providing feedback to the user based on the event.

20. The computing the location of the event relative to the wearable device; a repetitive pattern in the audio signal associated with the event; or 20. The system of claim 17, further configured to filter out incidents of an event based on at least one attribute of the audio signal associated with the event.

21. 1. A computer-implemented method for providing feedback to a user, comprising: Receiving ambient sound measurements measured using two or more microphones of a wearable device; determining an event based on the measured ambient sound; and determining a location attribute of the event relative to the wearable device; and providing feedback to the user based on the event and the location attribute.

22. 22. The computer-implemented method of claim 21, wherein the feedback comprises playing spatialized audio cues in addition to audio content that cancels the ambient sound.

23. determining the event based on the measured ambient sound; measuring the loudness of the sound level of the event; and and determining a location of the event relative to the wearable device using an audio signal associated with the event, wherein the location of the event is determined based on a direction and distance processed from the audio signal, and the location attributes of the event include at least one of the direction or the distance.

24. determining the event based on the ambient sound; the location of the event relative to the wearable device; a repetitive pattern in the audio signal associated with the event; or 24. The computer-implemented method of claim 23, further comprising filtering out incidents of an event based on at least one of the attributes of the audio signal associated with the event.

25. Determining the event based on the measured ambient sound includes:

24. The computer-implemented method of claim 23, comprising determining that the magnitude of the sound level of the event exceeds a threshold corresponding to the location of the event relative to the wearable device.

26. 26. The method of claim 25, wherein the threshold varies based on the location of the event relative to the wearable device, the threshold increasing as the location of the event gets closer to the user.

27. 1. A method for providing feedback to a user of a wearable device, comprising: Measuring ambient sound using two or more microphones on the wearable device; determining an event based on the measured ambient sound exceeding a threshold that varies among two or more different constituent events corresponding to two or more different situation criteria; and providing feedback to the user based on the determined event.

28. Each of the two or more different situation criteria is the threshold value associated with the determined event; a location attribute of the event relative to the wearable device; or 30. The method of claim 27, comprising customization configuration for at least one of the feedback to the user based on the determined event.

29. 30. The method of claim 28, wherein the customization configuration is input by the user.

30. Associating the customization configuration with the determined location attribute of the event relative to the wearable device; 30. The method of claim 29, further comprising identifying a pattern based on the user input and the association to determine a general rule for organization based on the location attribute.

31. 30. The method of claim 28, wherein the customized configuration comprises a default factory configuration.

32. 28. The method of claim 27, wherein each of the two or more different situation criteria includes a location attribute relative to the wearable device, and the location attribute determined based on the ambient sound is measured using the two or more microphones of the wearable device.

33. 33. The method of claim 32, wherein the threshold comprises a minimum or maximum distance corresponding to one of the two or more different situation criteria.

34. The method of claim 32 , wherein the threshold comprises a direction corresponding to one of the two or more different situation criteria.

35. The method of claim 32 , wherein each of the two or more different situation criteria corresponds to a physical location of the wearable device.

36. 28. The method of claim 27, wherein each of the two or more different situation criteria includes a sound class including at least one of speech, background sounds, or musical sequences.

37. 37. The method of claim 36, wherein the background sounds include alarms, footsteps, traffic noise, or sound patterns.

38. 38. The method of claim 37, wherein the threshold comprises a minimum sound level corresponding to one of the two or more different situation criteria.

39. 38. The method of claim 37, wherein the threshold value corresponds to a sensitivity of one of the two or more different situation criteria.

40. 30. The method of claim 27, wherein providing the feedback includes playing a recorded copy of the ambient sound to the user.

41. 41. The method of claim 40, wherein playing the recorded copy includes amplifying or attenuating the recorded copy of the ambient sound based on data collected by a feedback microphone in the wearable device.

42. 42. The method of claim 41, wherein amplifying or attenuating the recorded copy of the ambient sound is triggered by movement of the wearable device.

43. 43. The method of claim 42, wherein playing the recorded copy includes delaying playing the recorded copy based on the movement of the wearable device.

44. correlating the ambient sounds with the location attributes over a period of time; generating a saliency map mapping the variations in the ambient sound over the period of time at different locations; Measuring a movement record of the wearable device over the period of time; and 30. The method of claim 28, further comprising: determining a response of the user based on a relationship between the movement record and the saliency map.

45. Determining the event comprises: transmitting the measured ambient sound to a second device for processing; 28. The method of claim 27, further comprising: receiving the determined event and the corresponding feedback for providing to the user from the second device, the second device having greater processing capabilities than the wearable device.

46. 1. A system comprising: A wearable device having two or more microphones configured to measure ambient sound; a computing device connected to the wearable device, the computing device comprising: receiving the measured ambient sound from the wearable device; determining an event based on the ambient sound exceeding a threshold that varies among two or more different constituent events corresponding to two or more different situation criteria; and The system is configured to provide feedback to a user based on the event.

47. The wearable device, Evaluating a time requirement for locally determining an event based on the ambient sound exceeding the threshold; and 47. The system of claim 46, configured to transmit the measured ambient sound to the wearable device for determination when the time requirement exceeds a wait period.

48. Each of the two or more different situation criteria is the threshold value associated with the determined event; a location attribute of the event relative to the wearable device; or 47. The system of claim 46, comprising customization configuration for at least one of the feedback to the user based on the determined event.

49. 49. The system of claim 48, wherein the computing device is further configured to receive the customization configuration from the user, the user inputting the customization configuration on a graphical interface of the computing device.

50. 50. The system of claim 49, wherein the wearable device comprises a noise canceling audio device and the computing device comprises a smartphone.

51. A non-transitory computer readable medium storing instructions that, when executed by a wearable device, cause the wearable device to: measuring ambient sound using two or more microphones on the wearable device; determining an event based on the measured ambient sound exceeding a threshold that varies among two or more different constituent events corresponding to two or more different situation criteria; and A non-transitory computer-readable medium that causes providing feedback to a user based on the determined event.

Images