Adaptive ANC based on environmental trigger conditions

Abstract

This application relates to adaptive ANC based on environmental trigger conditions. A disclosed computer-implemented method can include applying sound cancellation via a sound reproduction system that reduces the amplitude of various sound signals. The method also includes identifying, in the sound signals, an external sound whose amplitude is to be reduced by the sound cancellation. The method then includes analyzing the identified external sound to determine whether the identified external sound is to be heard by a user, and upon determining that the external sound is to be heard by the user, the method includes modifying the sound cancellation such that the identified external sound is heard by the user. Various other methods, systems, and computer-readable media are also disclosed.

Description

[0001] This application is a continuation of application number 201880099008.4, filed December 20, 2018, having the title“Adaptive ANC based on environmental trigger conditions”.

[0002] Cross Reference to Related Applications

[0003] This application claims the benefit of U.S. Non-Provisional Application No. 16 / 171,389, filed October 26, 2018, the disclosure of which is incorporated by reference in its entirety. BACKGROUND

[0004] Active noise cancellation (ANC) is commonly used in headphones and other electronic devices to cancel out noise surrounding a user. For example, users often wear headphones with ANC on airplanes to drown out the noise of jet engines and remove sounds from nearby passengers. Active noise cancellation typically works by listening to external sounds and then generating a noise cancellation signal that is 180 degrees out of phase with the actual background noise. When the ANC signal and the external sound combine, the external sound is muted or at least greatly attenuated.

[0005] In a typical ANC application, a user will turn on the ANC function and leave it on until they take off the headset. For example, if a user is riding a mountain bike or a road bike, the user can wear ANC headphones or earbuds that allow the user to listen to music while completely muting or greatly attenuating external sounds. In such an example, the user will typically leave the ANC function running during the bike ride. However, during this ride, the user can miss some sounds that are important to the user, such as a car horn or a train whistle.

[0006] SUMMARY

[0007] As will be described in greater detail below, the present disclosure describes modifying active noise cancellation based on environmental trigger conditions. In situations where certain external noises should reach the user, embodiments herein can modify the active noise cancellation to allow those external sounds to pass through to the user. It should be noted that throughout this document, the term“noise cancellation,”“active noise cancellation,” or“sound cancellation” can refer to methods of reducing any type of audible noise or sound, respectively.

[0008] In one example, a computer-implemented method for modifying active noise cancellation based on environmental trigger conditions can include applying, via a sound reproduction system, noise cancellation that reduces the amplitude of various sound signals. The method can also include identifying, in the sound signals, an external sound whose amplitude is to be reduced by the active noise cancellation. The method can then include analyzing the identified external sound to determine whether the identified external sound is to be heard by a user, and upon determining that the external sound is to be heard by the user, the method can include modifying the active noise cancellation such that the identified external sound is heard by the user.

[0009] In some examples, modifying the active noise cancellation signal includes increasing the audibility of the identified external sound. Increasing the audibility of the identified external sound can include compressing the modified active noise cancellation signal such that the modified active noise cancellation signal is played back in a shortened timeframe. Additionally or alternatively, increasing the audibility of the identified external sound can include increasing the volume along a specified frequency band.

[0010] In some examples, the identified external sound can include various words, or a particular word or phrase. In some examples, the method can also include detecting from which direction the identified external sound originated, and presenting the identified external sound to the user as coming from the detected direction. In some examples, the active noise cancellation signal can be further modified to present subsequently occurring audio from the detected direction.

[0011] In some examples, a policy can be applied when it is determined that the external sound is to be heard by the user. In some examples, the identified external sound can be ranked according to a level of severity. In some examples, upon determining that the identified external sound has a minimum threshold level of severity, the active noise cancellation signal can be modified.

[0012] In some examples, the method for modifying active noise cancellation based on environmental trigger conditions can also include receiving an indication that an event has occurred within a specified distance of the user, and determining that the event is pertinent to the user. Then, based on the determination that the event is pertinent to the user, the active noise cancellation signal can be modified to allow the user to hear external sounds from the event site. In some examples, a microphone configured to listen for external sounds can be directionally oriented toward the event.

[0013] In some examples, the method can also include determining that another electronic device within a specified distance of the system has detected an external sound that is pertinent to the user. The method can then include determining a current location of the other electronic device, and physically or digitally orienting (i.e., beamforming) a microphone configured to listen for external sounds toward the determined location of the electronic device.

[0014] In some examples, modifying the active noise cancellation signal can include continuing to apply active noise cancellation to external sounds received from a plurality of locations while disabling active noise cancellation for external sounds received from a specified location. In some examples, modifying the active noise cancellation signal can include continuing to apply active noise cancellation to external sounds received from a particular person while disabling active noise cancellation for external sounds received from other people.

[0015] In some examples, modifying the active noise cancellation signal can include disabling active noise cancellation for a particular word detected in the external sounds while continuing to apply active noise cancellation for other words. For example, a listening user can be wearing an augmented reality (AR) headset, and an external user can say “barge in,” and the next phrase from the external user can be sent to the listening user while subsequent phrases from the external user are noise cancelled. In some examples, modifying the active noise cancellation signal can include temporarily pausing active noise cancellation and resuming active noise cancellation after a specified amount of time. In some examples, the sound reproduction system can further include a microphone for playing back the modified active noise cancellation signal to the user.

[0016] Further, a corresponding system for modifying active noise cancellation based on environmental trigger conditions can include several modules stored in memory, including a sound reproduction system configured to apply noise cancellation that reduces the amplitude of various noise signals. The system can also include an external sound identification module that identifies, among the noise signals, external sounds whose amplitudes are to be reduced by the noise cancellation. A sound analyzer can analyze the identified external sounds to determine whether the identified external sounds are to be heard by a user, and upon determining that an external sound is to be heard by the user, an ANC modification module can modify the noise cancellation so that the identified external sound is heard by the user.

[0017] In some examples, the above-described methods can be encoded as computer-readable instructions on a computer-readable medium. For example, the computer-readable medium can include one or more computer-executable instructions that, when executed by at least one processor of a computing device, can cause the computing device to apply, via a sound reproduction system, noise cancellation that reduces the amplitude of noise signals, identify, among the noise signals, external sounds whose amplitudes are to be reduced by the noise cancellation, analyze the identified external sounds to determine whether the identified external sounds are to be heard by a user, and upon determining that an external sound is to be heard by the user, modify the noise cancellation so that the identified external sound is heard by the user.

[0018] Features from any of the above mentioned embodiments can be used in combination with each other according to the general principles described herein. These and other embodiments, features and advantages will be more fully understood when the following detailed description is read in conjunction with the accompanying drawings and claims.

[0019] In particular, embodiments according to the application are disclosed in the appended claims relating to methods, systems and storage media, wherein any feature mentioned in one claim category (e.g. method) can also be claimed in another claim category (e.g. system, storage media and computer program product). The dependencies or back-references in the appended claims are chosen for formal reasons only. However, any subject matter resulting from intentional back-references (in particular multiple back-references) to any preceding claim, can also be claimed, such that any combination of claims and their features can be disclosed and claimed, not only the combinations explicitly given in the appended claims. The subject matter that can be claimed not only includes the combinations of features as set forth in the dependent claims, but also includes any other combination of the features of the claims, wherein each cited feature is cited in any claim category, in any combination with any other cited feature or combination of features. Furthermore, any of the embodiments and features described or depicted herein can be claimed in separate claims and / or in any combination with any other embodiment or feature described or depicted herein or in any combination with any feature of the appended claims.

[0020] In embodiments according to the application, one or more computer-readable non-transitory storage media can embody software that is operable when executed to perform a method according to the application or any of the above mentioned embodiments.

[0021] In embodiments according to the application, a system can comprise one or more processors; and at least one memory coupled to the processors and comprising instructions executable by the processors, the processors being operable when executing the instructions to perform a method according to the application or any of the above mentioned embodiments.

[0022] In embodiments according to the application, a computer program product, preferably comprising a computer-readable non-transitory storage media, can be operable when executed on a data processing system to perform a method according to the application or any of the above mentioned embodiments.

[0023] The present application provides the following:

[0024] 1. A computer-implemented method comprising:

[0025] applying sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals;

[0026] identifying, in the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation;

[0027] analyzing the identified external sound to determine whether the identified external sound is to be made audible to a user; and

[0028] upon determining that the external sound is to be made audible to the user, modifying the sound cancellation such that the identified external sound is made audible to the user.

[0029] 2). The computer-implemented method of 1), wherein modifying the sound cancellation signal further comprises increasing audibility of the identified external sound.

[0030] 3). The computer-implemented method of 2), wherein increasing audibility of the identified external sound comprises compressing the modified sound cancellation signal such that the modified sound cancellation signal is played back in a shortened time frame.

[0031] 4). The computer-implemented method of 2), wherein increasing audibility of the identified external sound comprises increasing volume along a specified frequency band.

[0032] 5). The computer-implemented method of 1), wherein the identified external sound comprises one or more words.

[0033] 6). The computer-implemented method of 1), further comprising:

[0034] detecting from which direction the identified external sound originates; and

[0035] presenting the identified external sound to the user as originating from the detected direction.

[0036] 7). The computer-implemented method of 6), further comprising further modifying the sound cancellation to present subsequently occurring audio from the detected direction.

[0037] 8). The computer-implemented method of 1), wherein one or more policies are applied when determining that the external sound is to be made audible to the user.

[0038] 9). The computer-implemented method of 1), wherein the identified external sound is ranked according to a level of severity.

[0039] 10). The computer-implemented method of 9), wherein the sound cancellation is modified upon determining that the identified external sound has a minimum threshold level of severity.

[0040] 11). A system comprising:

[0041] at least one physical processor;

[0042] physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:

[0043] apply sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals;

[0044] identify, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation;

[0045] analyze the identified external sound to determine whether the identified external sound is to be heard by a user; and

[0046] on a determination that the external sound is to be heard by the user, modify the sound cancellation so that the identified external sound is heard by the user.

[0047] 12). The system of 11), further comprising:

[0048] receiving an indication that an event occurred within a specified distance of a user; and

[0049] determining that the event is relevant to the user,

[0050] wherein the sound cancellation is modified based on the determination that the event is relevant to the user.

[0051] 13). The system of 12), further comprising orienting one or more microphones configured to listen for the external sound directionally toward a direction of the event.

[0052] 14). The system of 11), further comprising:

[0053] determining that another electronic device within a specified distance of the system has detected an external sound relevant to the user;

[0054] determining a current location of the other electronic device; and

[0055] orienting one or more microphones configured to listen for the external sound directionally toward the determined location of the electronic device.

[0056] 15). The system of 11), wherein modifying the sound cancellation comprises continuing to apply sound cancellation to external sounds received from a plurality of locations while disabling sound cancellation for external sounds received from a specified location.

[0057] 16). The system of 11), wherein modifying the sound cancellation includes continuing to apply sound cancellation to external sounds received from a particular person while disabling sound cancellation for external sounds received from other people.

[0058] 17). The system of 11), wherein modifying the sound cancellation includes disabling sound cancellation for particular words detected in the external sounds while continuing to apply sound cancellation to other words.

[0059] 18). The system of 11), wherein modifying the sound cancellation includes temporarily pausing sound cancellation and resuming sound cancellation after a specified amount of time.

[0060] 19). The system of 11), wherein the system further comprises a speaker for playing back the modified sound cancellation signal to the user.

[0061] 20). A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

[0062] apply sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals;

[0063] identify, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation;

[0064] analyze the identified external sound to determine whether the identified external sound is to be heard by a user; and

[0065] upon determining that the external sound is to be heard by the user, modify the sound cancellation so that the identified external sound is heard by the user.

[0066] 21). A computer-implemented method comprising:

[0067] applying sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals;

[0068] identifying, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation;

[0069] analyzing the identified external sound to determine whether the identified external sound is to be heard by a user; and

[0070] upon determining that the external sound is to be heard by the user, modifying the sound cancellation so that the identified external sound is heard by the user.

[0071] 22). The computer-implemented method of 21), wherein modifying the sound cancellation signal further comprises increasing audibility of the identified external sound.

[0072] 23). The computer-implemented method of 22), wherein increasing audibility of the identified external sound comprises compressing the modified sound cancellation signal such that the modified sound cancellation signal is played back in a shortened timeframe; and / or

[0073] wherein increasing audibility of the identified external sound comprises increasing volume along a specified frequency band.

[0074] 24). The computer-implemented method of any one of 21) to 23), wherein the identified external sound comprises one or more words.

[0075] 25). The computer-implemented method of any one of 21) to 24), further comprising:

[0076] detecting which direction the identified external sound originates from; and

[0077] presenting the identified external sound to the user as coming from the detected direction;

[0078] Optionally, the method further comprises further modifying the sound cancellation to present subsequently occurring audio from the detected direction.

[0079] 26). The computer-implemented method of any one of 21) to 25), wherein one or more policies are applied when determining that the external sound is to be made audible to the user.

[0080] 27). The computer-implemented method of any one of 21) to 26), wherein the identified external sound is ranked according to a level of severity;

[0081] Optionally, wherein the sound cancellation is modified upon determining that the identified external sound has a minimum threshold level of severity.

[0082] 28). A system comprising:

[0083] at least one physical processor;

[0084] physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:

[0085] apply sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals;

[0086] in the one or more sound signals, identifying an external sound whose amplitude is to be reduced by the sound cancellation;

[0087] analyzing the identified external sound to determine whether the identified external sound is to be let through to the user; and

[0088] on determining that the external sound is to be let through to the user, modifying the sound cancellation so that the identified external sound is let through to the user.

[0089] 29). The system of 28), further comprising:

[0090] receiving an indication that an event has occurred within a specified distance of the user; and

[0091] determining that the event is relevant to the user,

[0092] wherein the sound cancellation is modified based on the determination that the event is relevant to the user;

[0093] optionally, further comprising one or more microphones configured to listen for the external sound are directionally oriented towards a direction of the event.

[0094] 30). The system of 28) or 29), further comprising:

[0095] determining that another electronic device within a specified distance of the system has detected an external sound that is relevant to the user;

[0096] determining a current location of the other electronic device; and

[0097] directionally orienting one or more microphones configured to listen for the external sound towards the determined location of the electronic device.

[0098] 31). The system of any of 28) to 30), wherein modifying the sound cancellation comprises continuing to apply sound cancellation to external sounds received from a plurality of locations while disabling sound cancellation for external sounds received from a specified location.

[0099] 32). The system of any of 28) to 31), wherein modifying the sound cancellation comprises continuing to apply sound cancellation to external sounds received from a particular person while disabling sound cancellation for external sounds received from other people; and / or

[0100] wherein modifying the sound cancellation comprises disabling sound cancellation for a particular word detected in the external sound while continuing to apply sound cancellation for other words.

[0101] 33). The system of any one of 28) to 32), wherein modifying the sound cancellation comprises temporarily pausing sound cancellation, and resuming sound cancellation after a specified amount of time.

[0102] 34). The system of any one of 28) to 33), wherein the system further comprises a speaker for playing back the modified sound cancellation signal to a user.

[0103] 35). A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to perform the method of any one of 21) to 27), or cause the computing device to:

[0104] apply sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals;

[0105] identify, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation;

[0106] analyze the identified external sound to determine whether the identified external sound is to be heard by a user; and

[0107] on a determination that the external sound is to be heard by the user, modify the sound cancellation so that the identified external sound is heard by the user. BRIEF DESCRIPTION OF DRAWINGS

[0108] The accompanying drawings illustrate a number of example embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the present disclosure.

[0109] Figure 1 Embodiments of an artificial reality headset are shown.

[0110] Figure 2 Embodiments of an augmented reality headset and corresponding neckband are shown.

[0111] Figure 3 Embodiments of a virtual reality headset are shown.

[0112] Figure 4 A computing environment in which embodiments described herein can operate, including modifying active noise cancellation based on environmental trigger conditions, is shown.

[0113] Figure 5 A flowchart of an example method for modifying active noise cancellation based on environmental trigger conditions is shown.

[0114] Figure 6Alternative computing environments are shown in which active noise cancellation can be modified based on environmental trigger conditions.

[0115] Figure 7 Alternative computing environments are shown in which active noise cancellation can be modified based on environmental trigger conditions.

[0116] Figure 8 Alternative computing environments are shown in which active noise cancellation can be modified based on environmental trigger conditions.

[0117] Figure 9 Alternative computing environments are shown in which active noise cancellation can be modified based on environmental trigger conditions.

[0118] Figure 10 Alternative computing environments are shown in which active noise cancellation can be modified based on environmental trigger conditions.

[0119] Figure 11 Alternative computing environments are shown in which active noise cancellation can be modified based on environmental trigger conditions.

[0120] Throughout the drawings, like reference numerals and descriptions will be understood to indicate like, but not necessarily identical, elements. While the example embodiments described herein are amenable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail in this document. However, the example embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the appended claims.

[0121] DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0122] This disclosure relates generally to modifying active noise cancellation based on environmental trigger conditions. As will be explained in greater detail below, embodiments of the disclosure can determine that external sounds are important enough that they should be presented to a user even if the user has turned on noise cancellation. For example, a user can be in danger and a bystander can be shouting at the user to move. Embodiments described herein can determine that the shouts directed at the user are important to the user and that they should be presented to the user. Thus, embodiments herein can temporarily stop the noise cancellation process or can modify the noise cancellation signal so that the shouts (or other important sounds) reach the user. As noted above, active noise cancellation can be any type of operation that reduces a noise or sound signal. Thus, the terms "noise cancellation" and "sound cancellation" can be used synonymously herein.

[0123] In current active noise cancellation (ANC) implementations, ANC can be turned on and kept on. Traditional systems may lack the logic to determine whether ANC is applied. More precisely, the user simply turns the feature on, and ANC continues to run until it is turned off. Therefore, a user wearing ANC-enabled headphones may miss sounds that are important to them. For example, if a user is in a forest and a bear is roaring, a traditional ANC system might mute the bear's roar. In contrast, the embodiments described herein can determine that the bear's roar is important enough to the user that ANC should be deactivated or suppressed for a period of time. Furthermore, some words or phrases such as "Watch out!" or "Fire!" may be important enough that they should be presented to the user. Therefore, the embodiments described herein allow users to safely use ANC-enabled audio reproduction devices in a variety of different environments without worrying about missing important sounds.

[0124] Embodiments of this disclosure may include various types of artificial reality systems or combinations thereof. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include entirely generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect for the viewer). Furthermore, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof for purposes such as creating content in artificial reality and / or being used in artificial reality in other ways (e.g., to perform activities in artificial reality).

[0125] Artificial reality systems can be implemented in a wide variety of shape factors and configurations. Some artificial reality systems can be designed to operate without a near-eye display (NED), examples of which are... Figure 1 AR system 100. Other artificial reality systems may include NEDs that also provide visibility into the real world (e.g., Figure 2 AR systems (200) or NEDs that visually immerse users in artificial reality (e.g., Figure 3(VR system 300 in the text). While some artificial reality devices can be autonomous systems, others can communicate and / or collaborate with external devices to provide an artificial reality experience to the user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by the user, devices worn by one or more other users, and / or any other suitable external system.

[0126] Go to Figure 1 AR system 100 typically refers to a wearable device that is designed to fit the desired size around a part of the user's body (e.g., the head). Figure 1 As shown, system 100 may include a frame 102 and a camera component 104, the camera component 104 being coupled to the frame 102 and configured to collect information about the local environment by observing it. AR system 100 may also include one or more audio devices, such as output audio transducers 108(A) and 108(B) and an input audio transducer 110. Output audio transducers 108(A) and 108(B) may provide audio feedback and / or content to the user, and the input audio transducer 110 may capture audio in the user's environment.

[0127] As shown, AR system 100 may not necessarily include a NED located in front of the user's eyes. An AR system without an NED can take many forms, such as a headband, hat, hairband, belt, watch, wristband, ankle strap, ring, neckband, necklace, brooch, eyeglass frame, and / or any other suitable type or form of device. While AR system 100 may not include an NED, it may include other types of screens or visual feedback devices (e.g., a display integrated into one side of frame 102).

[0128] The embodiments discussed in this disclosure can also be implemented in AR systems that include one or more NEDs. For example, such as Figure 2 As shown, AR system 200 may include glasses device 202 with a frame 210 configured to hold a left display device 215(A) and a right display device 215(B) in front of a user's eyes. Display devices 215(A) and 215(B) may function together or independently to present an image or a series of images to the user. Although AR system 200 includes two displays, embodiments of this disclosure may be implemented in AR systems with a single NED or more than two NEDs.

[0129] In some embodiments, the AR system 200 can include one or more sensors, such as the sensors 240. The sensors 240 can generate measurement signals in response to motion of the AR system 200 and can be located substantially on any portion of the frame 210. The sensors 240 can include a position sensor, an inertial measurement unit (IMU), a depth camera assembly, or any combination thereof. In some embodiments, the AR system 200 can include or not include the sensors 240, or can include more than one sensor. In embodiments where the sensors 240 include an IMU, the IMU can generate calibration data based on the measurement signals from the sensors 240. Examples of the sensors 240 can include, but are not limited to, an accelerometer, a gyroscope, a magnetometer, other suitable types of sensors that detect motion, sensors for error correction of the IMU, or some combination thereof.

[0130] The AR system 200 can also include a microphone array having a plurality of acoustic sensors 220(A)-220(J) (collectively referred to as acoustic sensors 220). The acoustic sensors 220 can be transducers that detect changes in air pressure caused by acoustic waves. Each acoustic sensor 220 can be configured to detect sound and convert the detected sound into an electronic format (e.g., an analog or digital format). Figure 2 The microphone array in the AR system 200 can include, for example, ten acoustic sensors: 220(A) and 220(B), which can be designed to be placed within respective ears of a user; acoustic sensors 220(C), 220(D), 220(E), 220(F), 220(G), and 220(H), which can be positioned at different locations on the frame 210; and / or acoustic sensors 220(I) and 220(J), which can be positioned on respective neck bands 205.

[0131] The configuration of the acoustic sensors 220 of the microphone array can vary. Although the AR system 200 is shown in Figure 2 with ten acoustic sensors 220, the number of acoustic sensors 220 can be greater or less than ten. In some embodiments, using a higher number of acoustic sensors 220 can increase the amount of audio information collected and / or the sensitivity and accuracy of the audio information. Conversely, using a lower number of acoustic sensors 220 can reduce the computational power required by the controller 250 to process the collected audio information. Furthermore, the location of each acoustic sensor 220 of the microphone array can vary. For example, the location of the acoustic sensors 220 can include defined locations on the user, defined coordinates on the frame 210, an orientation associated with each acoustic sensor, or some combination thereof.

[0132] The acoustic sensors 220(A) and 220(B) can be positioned on different parts of the user's ear, such as behind the pinna or in the auricle or fossa. Alternatively, in addition to the acoustic sensors 220 inside the ear canal, there can be additional acoustic sensors on or around the ear. Positioning the acoustic sensors next to the user's ear canal can enable the microphone array to collect information about how sound reaches the ear canal. By positioning at least two of the acoustic sensors 220 on both sides of the user's head (e.g., as a binaural microphone), the AR device 200 can simulate binaural hearing and capture a 3D stereo sound field around the user's head. In some embodiments, the acoustic sensors 220(A) and 220(B) can be connected to the AR system 200 via a wired connection, and in other embodiments, the acoustic sensors 220(A) and 220(B) can be connected to the AR system 200 via a wireless connection (e.g., a Bluetooth connection). In still other embodiments, the acoustic sensors 220(A) and 220(B) can not be used in conjunction with the AR system 200 at all.

[0133] The acoustic sensors 220 can be positioned along the length of the temples, across the bridge, above or below the display devices 215(A) and 215(B), or some combination thereof, on the frame 210. The acoustic sensors 220 can be oriented so that the microphone array can detect sound in a wide range of directions around the user wearing the AR system 200. In some embodiments, an optimization process can be performed during manufacturing of the AR system 200 to determine the relative positions of each acoustic sensor 220 in the microphone array.

[0134] The AR system 200 can also include or be connected to external devices (e.g., companion devices), such as the neckband 205. As shown, the neckband 205 can be coupled to the eyewear device 202 via one or more connectors 230. The connectors 230 can be wired or wireless connectors, and can include electrical and / or non-electrical (e.g., structural) components. In some cases, the eyewear device 202 and the neckband 205 can operate independently without any wired or wireless connections between them. While the neckband 205 is shown as a separate device from the eyewear device 202, in some embodiments, the neckband 205 can be integrated into the eyewear device 202. Figure 2Components of the eyewear device 202 and the neckband 205 are shown in example locations on the eyewear device 202 and the neckband 205, but the components can be located elsewhere on the eyewear device 202 and / or the neckband 205 and / or distributed differently on the eyewear device 202 and / or the neckband 205. In some embodiments, components of the eyewear device 202 and the neckband 205 can be located on one or more additional peripheral devices that are paired with the eyewear device 202, the neckband 205, or some combination thereof. Furthermore, the neckband 205 generally represents any type or form of paired device. Accordingly, the following discussion of the neckband 205 can also apply to various other paired devices, such as a smartwatch, a smartphone, a wristband, other wearable devices, a handheld controller, a tablet computer, a laptop computer, etc.

[0135] Pairing an external device, such as the neckband 205, with an AR eyewear device can enable the eyewear device to achieve the form factor of a pair of glasses while still being able to provide sufficient battery and computing power for extended capabilities. Some or all of the battery power, computing resources, and / or additional features of the AR system 200 can be provided by the paired device or shared between the paired device and the eyewear device, thus generally reducing the weight, heat profile, and form factor of the eyewear device while still maintaining desired functionality. For example, the neckband 205 can allow components that would otherwise be included on the eyewear device to be included in the neckband 205 because a user can tolerate a heavier weight load on their shoulders than they would tolerate on their head. The neckband 205 can also have a larger surface area on which to spread and disperse heat to the surrounding environment. Thus, the neckband 205 can allow for greater battery and computing capacity than would otherwise be possible on a standalone eyewear device. Because the weight carried in the neckband 205 can be less jarring to a user than the weight carried in the eyewear device 202, a user can tolerate wearing a lighter eyewear device and carrying or wearing a paired device for a longer period of time than they would tolerate wearing a heavy standalone eyewear device, thus enabling a more immersive artificial reality environment into the user's daily activities.

[0136] The neckband 205 can be communicatively coupled with the eyewear device 202 and / or other devices. The other devices can provide certain functionality (e.g., tracking, localization, depth mapping, processing, storage, etc.) to the AR system 200. In Figure 2 In embodiments, the neckband 205 can include two acoustic sensors (e.g., 220(I) and 220(J)) that are part of a microphone array (or potentially form their own microphone subarray). The neckband 205 can also include a controller 225 and a power source 235.

[0137] Acoustic sensors 220(I) and 220(J) of neckband 205 can be configured to detect sound and convert the detected sound into an electronic format (analog or digital). In Figure 2 In embodiments, acoustic sensors 220(I) and 220(J) can be positioned on neckband 205 such that the distance between neckband acoustic sensors 220(I) and 220(J) and other acoustic sensors 220 positioned on eyewear device 202 is increased. In some cases, increasing the distance between acoustic sensors 220 of a microphone array can improve the accuracy of beamforming performed via the microphone array. For example, if a sound is detected by acoustic sensors 220(C) and 220(D), and the distance between acoustic sensors 220(C) and 220(D) is greater than, for example, the distance between acoustic sensors 220(D) and 220(E), the determined source location of the detected sound can be more accurate than if the sound was detected by acoustic sensors 220(D) and 220(E).

[0138] Controller 225 of neckband 205 can process information generated by sensors on neckband 205 and / or AR system 200. For example, controller 225 can process information from the microphone array describing sounds detected by the microphone array. For each detected sound, controller 225 can perform a DoA estimation to estimate the direction from which the sound arrived at the microphone array. Controller 225 can populate an audio dataset with this information as the microphone array detects sounds. In embodiments where AR system 200 includes an inertial measurement unit, controller 225 can compute all inertial and spatial calculations from the IMU located on eyewear device 202. Connector 230 can transfer information between AR system 200 and neckband 205 and between AR system 200 and controller 225. The information can be in the form of optical data, electrical data, wireless data, or any other transmittable data form. Moving the processing of information generated by AR system 200 to neckband 205 can reduce the weight and heat in eyewear device 202, making it more comfortable for the user.

[0139] Power source 235 in neckband 205 can provide power to eyewear device 202 and / or neckband 205. Power source 235 can include, but is not limited to, a lithium-ion battery, a lithium-polymer battery, a primary lithium battery, an alkaline battery, or any other form of power storage device. In some cases, power source 235 can be a wired power source. Including power source 235 on neckband 205 instead of on eyewear device 202 can help better distribute the weight and heat generated by power source 235.

[0140] As mentioned, some artificial reality systems can substantially replace one or more sensory perceptions of the real world with virtual experiences, rather than blending artificial reality with actual reality. One example of this type of system is a head-worn display system (e.g., VR system 300 in FIG. 1), which primarily or entirely covers a user’s field of view. VR system 300 can include a front rigid body 302 and a band 304 shaped to fit around a user’s head. VR system 300 can also include output audio transducers 306(A) and 306(B). In addition, while not shown in FIG. 1, front rigid body 302 can include one or more electronic elements, including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking emitters or detectors, and / or any other suitable devices or systems for creating an artificial reality experience. Figure 3 Figure 3

[0141] Artificial reality systems can include various types of visual feedback mechanisms. For example, the display devices in AR system 200 and / or VR system 300 can include one or more liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED (OLED) displays, and / or any other suitable type of display screen. Artificial reality systems can include a single display screen for both eyes, or can provide a display screen for each eye, which can provide additional flexibility for adjusting for accommodation or for correcting a user’s refractive error. Some artificial reality systems can also include optical subsystems with one or more lenses (e.g., traditional concave or convex lenses, Fresnel lenses, adjustable liquid lenses, etc.) through which a user can view a display screen.

[0142] In addition to or instead of using display screens, some artificial reality systems can include one or more projection systems. For example, the display devices in AR system 200 and / or VR system 300 can include micro-LED projectors (using, e.g., waveguides) that project light into the display devices, such as transparent combiner lenses that allow ambient light to pass through. The display devices can refract the projected light toward a user’s pupil and can enable a user to simultaneously view artificial reality content and the real world. Artificial reality systems can also be configured with any other suitable type or form of image projection system.

[0143] ​​The artificial reality system can also include various types of computer vision components and subsystems. For example, AR system 100, AR system 200, and / or VR system 300 can include one or more optical sensors, such as two-dimensional (2D) or three-dimensional (3D) cameras, time-of-flight depth sensors, single-beam or swept-frequency laser rangefinders, 3D LiDAR sensors, and / or any other suitable type or form of optical sensor. The artificial reality system can process data from one or more of these sensors to identify the location of the user, map the real world, provide context to the user about the real world surroundings, and / or perform various other functions.

[0144] The artificial reality system can also include one or more input and / or output audio transducers. In Figure 1 and Figure 3 In the example shown, output audio transducers 108(A), 108(B), 306(A), and 306(B) can include voice coil speakers, ribbon speakers, electrostatic speakers, piezoelectric speakers, bone conduction transducers, cartilage conduction transducers, and / or any other suitable type or form of audio transducer. Similarly, input audio transducer 110 can include a condenser microphone, a dynamic microphone, a ribbon microphone, and / or any other type or form of input transducer. In some embodiments, a single transducer can be used for both audio input and audio output.

[0145] Although not shown in Figures 1-3 , the artificial reality system can include a tactile (i.e., haptic) feedback system, which can be incorporated into headwear, gloves, tight clothing, handheld controllers, environmental devices (e.g., chairs, floor pads, etc.), and / or any other type of device or system. The tactile feedback system can provide various types of skin feedback, including vibration, force, traction, texture, and / or temperature. The tactile feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. The tactile feedback can be implemented using motors, piezoelectric actuators, fluidic systems, and / or various other types of feedback mechanisms. The tactile feedback system can be implemented independently of other artificial reality devices, within other artificial reality devices, and / or in conjunction with other artificial reality devices.

[0146] By providing tactile sensations, audible content, and / or visual content, artificial reality systems can create entire virtual experiences or enhance a user's real-world experiences in a variety of contexts and environments. For example, artificial reality systems can assist or extend a user's perception, memory, or cognition within a specific environment. Some systems can enhance a user's interaction with others in the real world or enable more immersive interaction with others in the virtual world. Artificial reality systems can also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government organizations, military organizations, commercial enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.), and / or for accessibility purposes (e.g., as hearing aids, visual aids, etc.). The embodiments disclosed herein can realize or enhance a user's artificial reality experience in one or more of these contexts and environments and / or in other contexts and environments.

[0147] Some AR systems can use a technique called Simultaneous Localization and Mapping (SLAM) to map the user's environment. SLAM mapping and location recognition technology can involve a variety of hardware and software tools that can create or update maps of the environment while simultaneously tracking the user's position within the mapped environment. SLAM can use many different types of sensors to create maps and determine the user's location within them.

[0148] SLAM technology can, for example, utilize optical sensors to determine a user's location. Wireless devices, including WiFi, Bluetooth, GPS, cellular, or other communication devices, can also be used to determine a user's location relative to a radio transceiver or transceiver group (e.g., a WiFi router or a GPS satellite group). Acoustic sensors, such as microphone arrays, or 2D or 3D sonar sensors, can also be used to determine a user's location within their environment. AR and VR devices (e.g., respectively) Figure 1 , Figure 2 and Figure 3 Systems 100, 200, and 300 can incorporate any or all of these types of sensors to perform SLAM operations, such as creating and continuously updating a map of the user's current environment. In at least some of the embodiments described herein, the SLAM data generated by these sensors can be referred to as "environmental data" and can indicate the user's current environment. This data can be stored in a local or remote data storage device (e.g., a cloud data storage device) and can be provided to the user's AR / VR device on demand.

[0149] When a user is wearing an AR headset or a VR headset in a given environment, the user can be interacting with other users or other electronic devices that are functioning as audio sources. In some cases, it can be desirable to determine where the audio sources are located relative to the user and then present the audio sources to the user as if they are coming from the location of the audio sources. The process of determining where the audio sources are located relative to the user can be referred to herein as "localization," and the process of reproducing playback of an audio source signal to appear as if it is coming from a particular direction can be referred to herein as "spatialization."

[0150] Localizing audio sources can be performed in a number of different ways. In some cases, an AR or VR headset can initiate a direction of arrival (DOA) analysis to determine the location of a sound source. The DOA analysis can include analyzing the intensity, spectrum, and / or time of arrival of each sound at the AR / VR device to determine the direction from which the sound originated. In some cases, the DOA analysis can include any suitable algorithm for analyzing the ambient acoustic environment in which the artificial reality device is located.

[0151] For example, the DOA analysis can be designed to receive input signals from microphones and apply a digital signal processing algorithm to the input signals to estimate the direction of arrival. These algorithms can include, for example, delay-and-sum algorithms, where the input signals are sampled and weighted and delayed versions of the resulting sampled signals are averaged together to determine the direction of arrival. Least mean squares (LMS) algorithms can also be implemented to create an adaptive filter. This adaptive filter can then be used to, for example, identify differences in signal strength or differences in time of arrival. These differences can then be used to estimate the direction of arrival. In another embodiment, the DOA can be determined by converting the input signals into the frequency domain and selecting particular bins within the time-frequency (TF) domain for processing. Each selected TF bin can be processed to determine whether that bin includes a portion of the audio spectrum with a direct path audio signal. Those bins with a portion of the direct path signal can then be analyzed to identify the angle at which the microphone array receives the direct path audio signal. The determined angle can then be used to identify the direction of arrival of the received input signal. Other algorithms not listed above can also be used to determine the DOA, either individually or in combination with the above algorithms.

[0152] In some embodiments, different users can perceive a sound source as coming from slightly different locations. This can be a result of each user having a unique head-related transfer function (HRTF), which can be determined by the user's anatomy including the positioning of the ear canal length and eardrum. Artificial reality devices can provide alignment and directional guides that a user can follow to customize the sound signals presented to the user based on their unique HRTF. In some embodiments, an artificial reality device can implement one or more microphones to listen for sounds within the user's environment. The AR or VR headset can use various different array transfer functions (e.g., any of the DOA algorithms identified above) to estimate the direction of arrival of the sound. Once the direction of arrival is determined, the artificial reality device can playback the sound to the user according to the user's unique HRTF. Thus, the DOA estimates generated using array transfer functions (ATF) can be used to determine the direction from which the sound will be played. The playback of the sound can be further improved based on how a particular user hears the sound according to the HRTF.

[0153] In addition to or as an alternative to performing DOA estimates, an artificial reality device can perform localization based on information received from other types of sensors. These sensors can include cameras, IR sensors, thermal sensors, motion sensors, GPS receivers, or in some cases sensors that detect the movement of a user's eyes. For example, as mentioned above, an artificial reality device can include an eye tracker or gaze detector that determines where a user is looking. A user's eyes will often look towards a sound source, even if only briefly. Such cues provided by a user's eyes can further assist in determining the location of a sound source. Other sensors such as cameras, thermal sensors, and IR sensors can also indicate the location of a user, the location of an electronic device, or the location of another sound source. Any or all of the above methods can be used individually or in combination to determine the location of a sound source, and can also be used to update the location of a sound source over time.

[0154] Some embodiments can implement the determined DOA to generate more customized output audio signals for the user. For example, an “acoustic transfer function” can characterize or define how sound is received from a given location. More specifically, an acoustic transfer function can define a relationship between parameters of a sound at its source location and parameters by which the sound signal is detected (e.g., by a microphone array or by a user’s ear). An artificial reality device can include one or more acoustic sensors that detect sound within a range of the device. A controller of the artificial reality device can estimate a DOA of the detected sound (e.g., using any of the methods identified above) and, based on parameters of the detected sound, can generate an acoustic transfer function specific to a location of the device. This customized acoustic transfer function can thus be used to generate a spatialized output audio signal in which the sound is perceived to come from a particular location.

[0155] Indeed, once the location of one or more sound sources is known, the artificial reality device can reproduce (i.e., spatialize) the sound signal to sound as if it came from the direction of that sound source. The artificial reality device can apply filters or other digital signal processing that changes the intensity, spectrum, or time of arrival of the sound signal. The digital signal processing can be applied in such a way that the sound signal is perceived to originate from the determined location. The artificial reality device can amplify or suppress certain frequencies or change the time of arrival of the signal to each ear. In some cases, the artificial reality device can create an acoustic transfer function specific to the location of the device and the direction of arrival of the detected sound signal. In some embodiments, the artificial reality device can reproduce the source signal in a stereo device or a multi-speaker device (e.g., a surround sound device). In such cases, a separate and distinct audio signal can be sent to each speaker. Each of these audio signals can be changed to sound as if they came from the determined location of the sound source according to the user’s HRTF and according to measurements of the location of the user and the location of the sound source. Thus, in this way, the artificial reality device (or speakers associated with the device) can reproduce the audio signal to sound as if it originated from a particular location.

[0156] A detailed description of how active noise cancellation can be modified based on environmental trigger conditions will be provided below with reference to Figures 4-11 For example, Figure 4The computing architecture 400 is shown in which many of the embodiments described herein can operate. The computing architecture 400 can include a computer system 401. The computer system 401 can include at least one processor 402 and at least some system memory 403. The computer system 401 can be any type of local or distributed computer system, including a cloud computer system. The computer system 401 can include program modules for performing various different functions. The program modules can be hardware-based, software-based, or can include a combination of hardware and software. Each program module can use or represent computing hardware and / or software to perform specified functions, including those described herein below.

[0157] For example, the communication module 404 can be configured to communicate with other computer systems. The communication module 404 can include any wired or wireless communication devices capable of receiving data from other computer systems and / or transmitting data to other computer systems. These communication devices can include a radio, such as a hardware-based receiver 405, a hardware-based transmitter 406, or a hardware-based transceiver capable of both receiving and transmitting data. The radio can be a WIFI radio, a cellular radio, a Bluetooth radio, a Global Positioning System (GPS) radio, or other types of radios. The communication module 404 can be configured to interact with databases, mobile computing devices (such as mobile phones or tablets), embedded systems, or other types of computing systems.

[0158] The computer system 401 can also include a microphone 407. The microphone 407 can be configured to listen to sounds outside of the computer system, including noise signals 419. These noise signals 419 can include any type of sound, including music, speech, conversations, street noise, or other forms of audio. In embodiments herein, substantially any type of audio data can be referred to as “noise” that will be filtered out using active noise cancellation. The noise cancellation can be performed by a noise cancellation module 409 of a sound reproduction module 408 in the computer system 401. The sound reproduction module 408 can be its own sound reproduction system, separate from the computer system 401, or can be a module within the computer system 401. The sound reproduction module 408 can generate a speaker signal that drives a speaker that is listened to by the user 416. For example, the sound reproduction module 408 can provide an audio signal to the user’s headphones or external speakers. The noise cancellation signal 417 generated by the noise cancellation module 409 can include the audio signal as well as a separate noise cancellation signal. These two signals are then combined so that the noise cancellation signal 417 cancels out the noise signal 419 and the user only hears the audio signal.

[0159] In addition, computer system 401 may include an external sound recognition module 410. The external sound recognition module 410 can identify one or more external sounds 411 within a noise signal 419. The noise signal may originate from an outdoor environment, an indoor environment, a crowded environment, or a largely deserted environment. The noise signal 419 may include spoken words or other sounds, such as sirens, car horns, shouts, etc., which may be important to user 416.

[0160] The sound analyzer 412 of the computer system 401 can analyze these external sounds 411 and determine 413 whether these sounds are important enough to interrupt active noise cancellation and present them to the user 416. If it is determined 413 to be yes, the ANC modification module 414 can directly modify the noise cancellation signal 415, or it can send an ANC modification instruction 418 to the noise cancellation module 409 so that it can generate a modified noise cancellation signal. The modified noise cancellation signal 415 can cause noise cancellation to stop completely, or it can cause noise cancellation to pause temporarily, or it can cause noise cancellation to be suppressed for a period of time. By modifying the noise cancellation signal in this way, the user 416 should be able to hear the external sounds 411 that are identified as important to the user. (See reference...) Figure 4 Method 400 and Figures 3-8 These embodiments will be described in more detail.

[0161] Figure 5 This is a flowchart of an exemplary computer implementation of a method 500 for modifying active noise cancellation based on environmental triggering conditions. Figure 5 The steps shown can be performed by any suitable computer executable code and / or computing system (including...). Figure 4 The system shown will be used to execute this. In one example, Figure 5 Each step shown can represent an algorithm whose structure includes multiple sub-steps and / or is represented by multiple sub-steps, examples of which will be provided in more detail below.

[0162] like Figure 5As shown, at step 510, one or more systems described herein can apply noise cancellation via a sound reproduction system that reduces the amplitude of one or more noise signals. For example, the sound reproduction module 408 of the computer system 401 can apply noise cancellation that reduces the amplitude of the noise signal 419. As mentioned above, the sound reproduction module 408 can be its own standalone system or device, or can be part of the computer system 401. The sound reproduction module 408 can include a noise cancellation module 409 that generates a noise cancellation signal 417 based on noise detected in the noise signal 419. For example, the microphone 407 on the computer system 401 can detect many different noise signals 419. These noise signals can include words, conversations, sounds from machines (including cars or airplanes), outdoor sounds, or other noises. Many of these noises can be unimportant to the user 416 and can be filtered out via the noise cancellation signal 417. However, in some cases, one or more sounds within the noise signal 419 can be important to the user.

[0163] The term "important" or "relevant" as used herein can refer to external sounds that can be interesting or useful or perhaps necessary for the safety of the user. Thus, sounds that are deemed relevant or important to the user can be any sounds that should be passed through to the user 416. Various types of logic, algorithms, machine learning, or other steps can be employed to determine which sounds are important to the user. For example, machine learning or neural networks can use various algorithms to recognize vocal patterns, vocal strain, tone of voice, particular words, particular users who are speaking, or to recognize other sound characteristics. Over time, millions of sounds can be recognized and categorized by the machine learning algorithm as potentially important to the user or as harmless. When such external sounds are recognized, the noise cancellation can be cancelled or modified so as to present the external sound 416 to the user.

[0164] The method 500 also includes identifying, in the noise signal 419, external sounds 411 whose amplitudes are to be reduced by the noise cancellation (step 520). As mentioned above, many different external sounds can be included in the noise signal 419. Each of these external sounds can be individually identified by the module 410 and analyzed by the sound analyzer 412 to determine whether the user 416 should hear the sound. Such sounds can include ambulance sirens, car horn sounds, people shouting, particular words or phrases (such as "stop" or "help"), animal noises including growls or barks, or other sounds that are important to the user.

[0165] In Figure 5At step 530, the sound analyzer 412 can analyze the identified external sound 411 to determine whether the identified external sound is to be heard by the user 416 (step 530). If the sound analyzer 412 determines that the sound is not available to the user, then the noise cancellation continues. If the sound analyzer 412 determines that the external sound is to be heard by the user 416, then the ANC modification module 414 can modify the noise cancellation such that the identified external sound is heard by the user (step 540). The ANC modification module 414 can modify the noise cancellation signal 415 such that the identified external sound 411 is heard by the user. The ANC modification can include reducing the level of active noise cancellation, temporarily pausing active noise cancellation, or completely turning off the ANC.

[0166] In some embodiments, modifying the active noise cancellation signal 415 can include increasing the audibility of the identified external sound. For example, if the identified external sound 411 is important enough to modify or remove the ANC, embodiments herein can take additional steps to ensure that the external sound 411 is heard more clearly. One such step can be to increase the volume of the external sound such that it is more easily heard by the user 416. Additionally or alternatively, the ANC modification module can increase the audibility of the identified external sound by compressing the modified active noise cancellation signal such that the modified active noise cancellation signal is played back in a shortened timeframe. The shortened playback can provide the external sound 411 in the form of a short burst that is quickly recognizable to the user. In other cases, increasing the audibility of the identified external sound can include increasing the volume along a specified frequency band. For example, if the external sound 411 is a spoken word or series of words, the frequencies within the frequency band from approximately 300 Hz to 3000 Hz can be amplified to provide greater volume to the spoken words. Other non-amplified frequencies can also be attenuated to provide even greater clarity to the spoken words.

[0167] In some embodiments, the identified external sound 411 can be a specific word or phrase. For example, as shown in the computing environment 600, a speaking user 608 can speak a specific word 602 that is detected by a microphone 606 of a sound reproduction system 604. A sound analyzer 607 of the sound reproduction system 604 can determine that the specific word 602 (e.g., “move!”) is a word that is relevant to the user 601. Accordingly, the ANC module 605 can modify the active noise cancellation such that the word 602 reaches the user 601. Figure 6

[0168] ​Similarly, if a user or a group of speaking users (e.g., 609) speak a word phrase 603 that is relevant to user 601, the sound analyzer 607 can detect the word phrase, and the ANC module 605 can modify the active noise cancellation to allow the word phrase 603 to reach the user 601. In some embodiments, a list of specific words or word phrases can be stored in a data store, either locally to the sound reproduction system 604 or remotely from the sound reproduction system 604. The list of words or phrases can include words or phrases that are relevant to the user 601. The list can be edited or updated by the user 601. Alternatively, the list can be universal to all users. In other cases, the list of words or phrases can be dynamic, such that specific words or phrases can be more important to a user in certain situations or at certain locations, while at other locations the words can be safely muted by active noise cancellation. The policy 420 can be used to determine when certain words or phrases are delivered to the user 601.

[0169] In certain cases, modifying the ANC can include disabling active noise cancellation for specific words detected in the external sound, while continuing to apply active noise cancellation for other words. For example, if a speaking user 608 is providing a continuous stream of words, the sound analyzer 607 can identify certain words to be delivered to the user 601, and certain words to be cancelled via noise cancellation. Accordingly, the ANC module 605 of the sound reproduction system 604 can disable or temporarily pause active noise cancellation, and then resume active noise cancellation after a specified amount of time (e.g., after the words 602 have been played back to the user). In some examples, the modified ANC signal can be played back to the user 601 via a speaker built into the sound reproduction system 604, or a speaker signal can be sent to a speaker or headset connected to the sound reproduction system.

[0170] Figure 7 Embodiments are shown that identify specific natural or artificial sounds and provide them to the user 601. The sound analyzer 607 of the sound reproduction system 604 can continuously or continuously analyze the sound picked up by the microphone 606. Upon determining that an external sound is sufficiently important to the user 601, the ANC module 605 can modify the audio output to the user 601, thereby modifying or removing active noise cancellation. For example, when the sound analyzer 607 detects a siren 610 from an ambulance 613, fire truck, police car, or other emergency vehicle, the ANC module can modify the active noise cancellation such that the siren 610 is delivered to the user with substantially no noise cancellation (and possibly with some acoustic enhancement to make the siren louder and clearer).

[0171] Similarly, if the user 601 is outside and hears a growl 611 of a bear 614 or a hissing sound from a snake or other animal sound that is important to the user, the ANC module can modify the active noise cancellation such that the user 601 hears the growl 611 of the bear or other sound. Further, if a person 615 is shouting 612 or crying or screaming, the tone, pitch, or stress of the shouting 612 can be analyzed to indicate that the person is in distress or perhaps angry at the user 601. The sound analyzer 607 can indicate to the ANC module that the shouting 612 is serious and will be passed to the user 601. In some cases, the sound reproduction system 604 can internally rank the identified external sounds according to a level of seriousness. Thus, for example, the seriousness of a growl 611 of a bear can rank ahead of a siren 610, or the seriousness of a person’s shouting can be ranked more toward the front depending on their words or level of distress. In this way, the active noise cancellation can be modified based on the level of urgency or seriousness of the external sound. In certain cases, the active noise cancellation is modified as long as there is a minimum level of seriousness associated with the external sound.

[0172] Figure 8 Embodiments are shown in which the sound reproduction system 604 includes a direction analyzer 620. The direction analyzer 620 can be configured to detect from which direction the identified external sound 622 originated. For example, the direction analyzer can analyze the signal strength of the sound 622 and determine that the signal is strongest in the direction 621. Other means of determining the direction of the identified sound 622 can also be used, including receiving a location indication from another electronic device. Once the direction 621 is determined, the ANC module 605 can use the direction to modify the identified external sound and present it to the user 601 as coming from the detected direction 621. Thus, the modified ANC signal 623 can include audio processing that makes the modified signal sound as if it is coming from the direction 621. In some cases, the active noise cancellation signal 623 can be further modified to present subsequently occurring audio as if it is coming from the detected direction. Thus, once the source of the external sound 622 is identified, future external sounds from that source can be presented to the user 601 as if they are coming from that source, regardless of whether the user moves or reorients their body.

[0173] Figure 9One embodiment is shown in which active noise cancellation can be modified based on receiving an indication 634 that an event occurred within a specified distance 633 of the user 601 and that the event is relevant to the user. For example, a building 632 can be on fire at a distance from the general location of the user 601. The event analyzer 630 can determine where the event occurred from the information in the event indication 634. The sound reproduction system 604 can include a GPS, WiFi, Bluetooth, cellular radio, or other radio that can be used to determine its own location. Thus, using the location of the sound reproduction system 604 and the location of the event (e.g., the building 632), the event analyzer 630 can determine the distance 633 to the event. If the user 601 is close enough to the event, the ANC signal 631 can be modified to pass through the sound from the direction of the event. If the distance 633 is too far, the event analyzer 630 can determine that the event is not relevant enough to the user and active noise cancellation can continue uninterrupted. Further, even if the event is close enough to the user, the event analyzer 630 can determine that the event is not relevant to the user. Thus, in this case, the audio from the direction of the event can continue to be filtered out by active noise cancellation. As with the list of words or phrases, the user 601 can specify which events are important to that user and which events should interrupt active noise cancellation.

[0174] In some cases, the user 601 can be out walking or running, or riding a bicycle or scooter. As such, the user can pass by a number of different events. For each event that is determined to be relevant to the user, the ANC module 605 can modify the active noise cancellation signal to allow the user 601 to hear the outside sound from the event site. In some embodiments, the microphone 606 that is configured to listen to outside sounds can be directionally oriented toward the direction of the event. Thus, the microphone itself can be adjusted or actuated to a new position to more clearly capture the audio from the event. Alternatively, electronic sound processing can be implemented to directionally focus the microphone 606 on the sound from the event.

[0175] In some embodiments, different types of electronic equipment (in addition to microphones) can be used to detect the occurrence of events in the vicinity of the user. For example, optical sensors including cameras, rangefinders, LiDAR, sonar, or other optical sensors can be used to detect the occurrence of events. Other sensors can include infrared sensors, temperature sensors, motion sensors, or other sensors that can be configured to identify events that can be important to the user. As with audio input, the event analyzer 630 can be configured to analyze the camera or other sensor input to detect when an event occurs. The event analyzer 630 can then determine whether the event is sufficiently relevant to the user. If the event is sufficiently relevant to the user, then noise cancellation can be interrupted to allow the user to hear ambient audio. If the event is not sufficiently relevant, then active noise cancellation can continue without interruption. Still further, as with the list of words or phrases, the user 601 can specify which events detected by the camera or other sensors are important to the user, and which events should interrupt active noise cancellation.

[0176] Figure 10 Embodiments are shown in which multiple sound detection and reproduction systems are in the same relative position. These sound detection and reproduction systems can use WiFi, Bluetooth, or any of the other wireless radios described above to communicate with each other. The sound detection and reproduction systems 604A / 604B can indicate to each other that an event has occurred that is relevant to the user and should be heard. For example, the sound detection and reproduction system 604A can determine that another electronic device within the system's specified distance has detected an external sound that is relevant to the user. The sound detection and reproduction system 604B can send an indication 642 of the relevant sound to the sound detection and reproduction system 604A, for example. The sound detection and reproduction system 604A can then determine its current positioning and the current positioning of the other electronic device. The sound detection and reproduction system 604A can then orientally orient its microphone toward the sound detection and reproduction system 604B to listen for external sounds from the direction of the sound detection and reproduction system 604B.

[0177] Accordingly, for example, group 640 can emit a sound 641 near sound detection and reproduction system 604B. Microphone 606B can detect the sound 641 and use sound analyzer 607B to determine whether the sound is noteworthy and relevant to other users. Sound detection and reproduction system 604B can then broadcast an indication 642 of the relevant sound to sound detection and reproduction system 604A, as well as other systems or electronic devices. Each sound detection and reproduction system can then use its own sound analyzer (e.g., 607A) to determine, respectively, whether the sound is relevant and should be presented to the user. The microphones can be oriented directionally toward the location of sound detection and reproduction system 604B, or toward a location identified by sound detection and reproduction system 604B. The ANC module (e.g., 605A / 605B) of each sound detection and reproduction system can then modify the ANC signal accordingly, or not.

[0178] In some embodiments, each sound detection and reproduction system can be connected to an augmented reality (AR) headset (e.g., 100 or 200 of FIG. 1, respectively) or a portion thereof, or to a virtual reality (VR) headset (e.g., 300 of FIG. 1) or a portion thereof. These headsets can be worn by users in a common room or building. Each of these headsets can communicate their current location within the room or building (or outdoor area) to the other headsets. Other communications can include the indication 642 of a relevant sound. Accordingly, in this case, one AR headset can detect a relevant sound (e.g., someone shouting) and can broadcast an indication of the sound to the other headsets in the room, building, or outdoor area. The headset (and corresponding sound reproduction system) of each user can then determine, according to the above embodiments, whether the sound is relevant to that user and whether to modify the ANC for that user. Figure 1 or Figure 2 In some embodiments, each sound detection and reproduction system can be connected to an augmented reality (AR) headset (e.g., 100 or 200 of FIG. 1, respectively) or a portion thereof, or to a virtual reality (VR) headset (e.g., 300 of FIG. 1) or a portion thereof. These headsets can be worn by users in a common room or building. Each of these headsets can communicate their current location within the room or building (or outdoor area) to the other headsets. Other communications can include the indication 642 of a relevant sound. Accordingly, in this case, one AR headset can detect a relevant sound (e.g., someone shouting) and can broadcast an indication of the sound to the other headsets in the room, building, or outdoor area. The headset (and corresponding sound reproduction system) of each user can then determine, according to the above embodiments, whether the sound is relevant to that user and whether to modify the ANC for that user. Figure 3 or a portion thereof. These headsets can be worn by users in a common room or building. Each of these headsets can communicate their current location within the room or building (or outdoor area) to the other headsets. Other communications can include the indication 642 of a relevant sound. Accordingly, in this case, one AR headset can detect a relevant sound (e.g., someone shouting) and can broadcast an indication of the sound to the other headsets in the room, building, or outdoor area. The headset (and corresponding sound reproduction system) of each user can then determine, according to the above embodiments, whether the sound is relevant to that user and whether to modify the ANC for that user.

[0179] Figure 11 One embodiment is shown in which ANC module 605 modifies the active noise cancellation signal to continue applying active noise cancellation to external sounds received from one person while disabling active noise cancellation for external sounds received from another person. In Figure 11 In this case, user 650 can be speaking in audio output 652 while user 651 can be speaking in audio output 653. Sound analyzer 607 can determine that audio output 653 is to be passed to user 601 based on a policy or based on tone or sound tension, while ANC is to continue to be applied to audio output 652 from user 650.

[0180] In some cases, a policy might instruct that friends or family members should be prioritized, or that users who scream or shout should be prioritized. For example, computer system 401 can access user 416's contact list or social media accounts. Such a contact list or social media account can indicate who the user's family or friends are. If voice analyzer 412 identifies such family members or friends, computer system 401 can access policies regarding ANC for friends and family. Policies or settings (e.g., Figure 4 The settings 420 can instruct, for example, that ANC will be automatically turned off or reduced when a friend or family member speaks to user 416. Other policies may specify how to control ANC when someone shouts or a specific word is detected. These ANC policies and settings 420 can be stored in computer system 401 or in remote data storage such as cloud data storage. Computer system 401 can access these policies each time a decision is made on whether to use ANC or not. Regardless of the policy decision made, sound analyzer 607 can determine that audio output 653 from user 651 will be played back to user 651 before audio output 652 is played back to user 601. In this case, audio output 652 can be stored in data storage and played back to user 601 later.

[0181] In a similar manner, the sound reproduction system 604 can determine that external sounds from a particular location are more important than sounds from another location. In this case, the ANC module 605 can modify the active noise cancellation signal to continue applying active noise cancellation to external sounds received from some locations, while disabling or reducing active noise cancellation for external sounds received from a specific location. Thus, for example, even in a large city where sound may be received from all directions, the sound reproduction system 604 can be configured to point the microphone in a specific direction and apply noise cancellation to sounds received from other directions.

[0182] Furthermore, a corresponding system for modifying active noise cancellation based on environmental triggering conditions may include several modules stored in memory, including a sound reproduction system configured to apply noise cancellation that reduces the amplitude of various noise signals. The system may also include an external sound recognition module that identifies external sounds in the noise signal whose amplitude will be reduced by noise cancellation. A sound analyzer can analyze the identified external sounds to determine whether the identified external sounds should be heard by the user, and if it is determined that the external sounds should be heard by the user, the ANC modification module can modify the noise cancellation so that the identified external sounds are audible to the user.

[0183] In some examples, the above-described methods can be encoded as computer-readable instructions on a computer-readable medium. For example, the computer-readable medium can include one or more computer-executable instructions that, when executed by at least one processor of a computing device, can cause the computing device to apply noise cancellation that reduces an amplitude of a noise signal via a sound reproduction system, identify, in the noise signal, an external sound whose amplitude is to be reduced by the noise cancellation, analyze the identified external sound to determine whether the identified external sound is to be let through to a user, and modify the noise cancellation such that the identified external sound is let through to the user when it is determined that the external sound is to be let through to the user.

[0184] Thus, using the embodiments herein, a user can use active noise cancellation with confidence in a variety of different environments, knowing that if an important sound passes by, they will not miss it. The systems herein can determine that a sound important to the user has been received, and can temporarily stop or suppress active noise cancellation to allow the important sound to be heard by the user. Such embodiments can keep the user safe and aware of events happening around them, even when the user is wearing an active noise cancellation headset.

[0185] As detailed above, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing devices can each include at least one memory device and at least one physical processor.

[0186] In some examples, the term“memory device” generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device can store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, without limitation, random access memory (RAM), read only memory (ROM), flash memory, hard disks, solid-state drives (SSDs), optical disks, caches, variations or combinations of one or more of these devices, or any other suitable storage memory.

[0187] In some examples, the term "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor can access and / or modify one or more modules stored in one of the above-described memory devices. Examples of physical processors include, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) implementing softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the aforementioned components, variations or combinations of one or more of the aforementioned components, or any other suitable physical processors.

[0188] Although illustrated as separate elements, the modules described and / or illustrated herein can represent portions of a single module or application. In addition, one or more of these modules can represent one or more software applications or programs, which when executed by a computing device, can cause the computing device to perform one or more tasks. For example, one or more of the modules described and / or illustrated herein can represent modules stored and configured to run on one or more of the computing devices or systems described and / or illustrated herein. One or more of these modules can also represent all or portions of one or more specialized computers configured to perform one or more tasks.

[0189] In addition, one or more of the modules described herein can transform data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the modules described herein can receive data to be transformed, transform the data, output the results of the transformation to perform a function, use the results of the transformation to perform the function, and store the results of the transformation to perform the function. Additionally or alternatively, one or more of the modules described herein can transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form to another by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0190] In some embodiments, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, without limitation, transmission-type media (e.g., carrier waves) and non-transitory media, such as, for example, magnetic storage media (e.g., hard disk drives, tape drives, and floppy disks), optical storage media (e.g., Compact Discs (CDs), Digital Video Discs (DVDs), and BLU-RAY discs), electronic storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0191] Embodiments of the present disclosure can include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which can include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and / or derivatives thereof. Artificial reality content can include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content can include video, audio, touch feedback, or some combination thereof, any of which can be presented in a single channel or in multiple channels (such as stereo videos that produce a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality can also be associated with applications, products, accessories, services, or some combination thereof, that are used to create content in an artificial reality and / or used in conjunction with the artificial reality (such as perform activities in the artificial reality), which can be performed in a physical, augmented, or mixed reality setting. The artificial reality system that provides the artificial reality content can be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0192] The order of the process parameters and steps described and / or illustrated herein is merely exemplary and can be varied as desired. For example, while steps illustrated and / or described herein can be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated herein can also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0193] The preceding description is provided to enable any person skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. The exemplary description is not intended to be exhaustive or to be limited to any precise forms disclosed. Many modifications and variations are possible in light of this disclosure without departing from the spirit or scope of the disclosure. The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. In determining the scope of the disclosure, the appended claims should be accorded with the broadest construction possible, in conformity with the general principles set forth by the United States Supreme Court in the case of M.A.N. v. Teague, 307 U.S. 244, 59 S. Ct. 880, 83 L. Ed. 1331 (1939).

[0194] Unless otherwise noted, the terms "connected to" and "coupled to" (and their derivatives), as used in the specification and claims, are to be construed as permitting both direct and indirect (i.e. via another element or component) connection. In addition, the terms "a" and "an," as used in the specification and claims, are to be construed as meaning "at least one of." Finally, for ease of use, the terms "including" and "having" (and their derivatives), as used in the specification and claims, are intended to be equivalent to the word "comprising."

Claims

1. A computer-implemented method comprising: applying sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals; identifying, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation, the identified external sound originating from a determined direction; analyzing the identified external sound using machine learning to adaptively learn which identified external sounds are to be heard by a user, the machine learning configured to determine whether an identified external sound is to be heard by the user based on one or more sounds or sound characteristics of the external sound; and upon determining that the external sound is to be heard by the user, modifying the sound cancellation so that the identified external sound is heard by the user, the modifying including presenting the identified external sound to the user as coming from the determined direction.

2. The computer-implemented method of claim 1, wherein, The modifying the sound cancellation further includes increasing audibility of the identified external sound.

3. The computer-implemented method of claim 2, wherein, The increasing audibility of the identified external sound includes compressing a modified sound cancellation signal so that the modified sound cancellation signal is played back in a shortened timeframe.

4. The computer-implemented method of claim 2, wherein, The increasing audibility of the identified external sound includes increasing volume along a specified frequency band.

5. The computer-implemented method of claim 1, wherein, The identified external sound includes one or more spoken words.

6. The computer-implemented method of claim 1, wherein The modifying the sound cancellation includes temporarily pausing sound cancellation and resuming sound cancellation after a specified amount of time.

7. The computer-implemented method of claim 1, further comprising further modifying the sound cancellation to present subsequently occurring audio from the detected direction.

8. The computer-implemented method of claim 1, wherein, Applying one or more policies when it is determined that the external sound is to be heard by the user.

9. The computer-implemented method of claim 1, wherein, Ranking the identified external sound according to a level of severity.

10. The computer-implemented method of claim 9, wherein, Modifying the sound cancellation upon determining that the identified external sound has a minimum threshold level of severity.

11. A system comprising: at least one physical processor; physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to: apply sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals; identify, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation, the identified external sound originating from a determined direction; analyze the identified external sound using machine learning to adaptively learn which identified external sounds are to be heard by a user, the machine learning configured to determine whether an identified external sound is to be heard by the user based on one or more sounds or sound characteristics of the external sound; and upon determining that the external sound is to be heard by the user, modify the sound cancellation so that the identified external sound is heard by the user, the modifying including presenting the identified external sound to the user as coming from the determined direction.

12. The system of claim 11, further comprising: receiving an indication that an event occurred within a specified distance of a user; and determining that the event is relevant to the user, wherein the sound cancellation is modified based on the determination that the event is related to the user.

13. The system of claim 12, further comprising one or more microphones configured to listen for the external sound are directionally oriented toward the event.

14. The system of claim 11, further comprising: determining that another electronic device within a specified distance of the system has detected an external sound related to the user; determining a current location of the other electronic device; and directionally orienting one or more microphones configured to listen for the external sound toward the determined current location of the other electronic device. modifying the sound cancellation includes continuing to apply sound cancellation to external sounds received from multiple locations while disabling sound cancellation for external sounds received from a specified location.

15. The system of claim 11, wherein, modifying the sound cancellation includes continuing to apply sound cancellation to external sounds received from a particular person while disabling sound cancellation for external sounds received from other people.

16. The system of claim 11, wherein, modifying the sound cancellation includes disabling sound cancellation for particular words detected in the external sound while continuing to apply sound cancellation to other words.

17. The system of claim 11, wherein, modifying the sound cancellation includes temporarily pausing sound cancellation and resuming sound cancellation after a specified amount of time.

18. The system of claim 11, wherein, the system further comprises a speaker for playing back the modified sound cancellation signal to the user.

19. The system of claim 11, wherein, 20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: apply sound cancellation via a sound reproduction system, the sound cancellation reducing an amplitude of one or more sound signals; identify, among the one or more sound signals, an external sound whose amplitude is to be reduced by the sound cancellation, the identified external sound originating from a determined direction; analyze the identified external sound using machine learning to adaptively learn which identified external sounds are to be heard by a user, the machine learning configured to determine whether an identified external sound is to be heard by the user based on one or more sounds or sound characteristics of the external sound; and upon determining that the external sound is to be heard by the user, modify the sound cancellation such that the identified external sound is heard by the user, the modification including presenting the identified external sound to the user as originating from the determined direction. ​

Images