User attention indicator

Abstract

A method for visualizing sound audibility of an external audio signal, the method comprising: acquiring at least one external audio signal; obtaining at least one of: the internal audio signal, and an estimate of the at least one internal audio signal; estimating external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and an estimate of the at least one internal audio signal; and generating at least one visualization based on the estimated audibility of the external sound, such that the visualization provides an indication of the audibility of the external audio source.

Description

Technical Field

[0001] The present application relates to apparatus and methods for indicating user attention. Background Art

[0002] User attentiveness is often difficult to determine. In many cases, it is difficult to determine when another person operating a device can hear someone trying to get their attention.

[0003] For example, immersive communications and audio scenes can significantly attract the interest of users of mobile handsets that implement the immersive voice and audio service (IVAS) audio codec, allowing users to be almost completely immersed in the audio scene rendered to them. Even if the device (e.g., a mobile handset with a microphone) is configured to deliver audio signals to the user, the user's cognitive abilities can be immersed in the communication without "hearing" the local speaker.

[0004] Similarly, when the device is a vehicle in which the user is seated, the qualities of noise cancellation and noise blocking implemented as part of the vehicle's noise vibration and sound quality design can significantly acoustically isolate the user or passenger of the vehicle and prevent people outside the vehicle from communicating with the driver or passenger.

[0005] Additionally, many motorcyclists wear noise cancelling or noise blocking devices inside their helmets in an attempt to reduce hearing damage caused by wind noise generated by the helmet. However, these noise cancelling or noise blocking devices can cause problems because someone who attracts the attention of the motorcyclist may not attract the attention of the rider.

[0006] In addition, it is known that headphones equipped with ANC (active noise cancellation) and pass-through / transparency features are becoming more and more common. ANC actively (using electronics, microphones and speaker elements) attenuates sounds from external sound sources for the headphone user. The pass-through / transparency mode, in turn, actively plays external sound sources to the user wearing the headphone so that the user can hear the sounds of the surrounding environment (such as cars) and hear and talk to other people in the same space. Ideally, in transparency mode, the user can hear the sounds of the surrounding environment as if they were not wearing headphones.

[0007] Many headsets allow users to choose how much external sound they hear, so a device could gradually switch between ANC and Transparency modes.

[0008] The headset user can also control how loudly the headset plays music and other internal sounds from the device connected to the headset. "Internal sound" hereinafter is used to cover all internal sound sources, such as the device playing music, telecommunications, UI sounds, any application sounds, etc. In the following examples, the example "internal sound" is a music source, but it can be any other suitable source.

[0009] There are two different routes that a headphone user can typically hear external sound. First, a transparent mode can be configured to actively play external sound, and second, external sound can acoustically leak through the headphone and around the headphone into the user's ears. In the following disclosure, pure acoustic leakage is referred to as acoustic leakage, and the combination of acoustic leakage and transparent mode is referred to as leakage. Summary of the invention

[0010] According to a first aspect, a method for visualizing the audibility of sound of an external audio signal is provided, the method comprising: obtaining at least one external audio signal; obtaining at least one of the following: an internal audio signal, and an estimate of at least one internal audio signal; estimating the external sound audibility based at least in part on the at least one external audio signal and at least one of the following: the internal audio signal, and the estimate of at least one internal audio signal; and generating at least one visualization based on the estimated external sound audibility so that the visualization provides an indication of the audibility of the external audio source.

[0011] Acquiring at least one external audio signal may include acquiring at least one external microphone audio signal, wherein the at least one external microphone is located on or acoustically coupled to an external surface of the device such that the at least one external microphone audio signal is configured to capture audio external to the device.

[0012] Acquiring at least one internal audio signal may include acquiring at least one internal microphone audio signal, wherein the at least one internal microphone is located on or acoustically coupled to an interior surface of the device such that the at least one internal microphone audio signal is configured to capture audio inside the device.

[0013] Obtaining an estimate of at least one internal audio signal may include estimating at least one internal audio signal to be output by a transducer within the device, such that the estimate of the at least one internal audio signal is configured to assist in estimating the external sound audibility.

[0014] Estimating the audibility of external sounds based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of the at least one internal audio signal may include: determining an acoustic leakage estimate based on the at least one external audio signal and a function defining a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; and generating the at least one external sound audibility based on subtracting the anti-noise audio signal from the acoustic leakage estimate.

[0015] Estimating the audibility of external sounds based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of the at least one internal audio signal may include: determining an acoustic leakage estimate based on the at least one external audio signal and a function defining a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; and generating the at least one external sound audibility based on subtracting the anti-noise audio signal from the acoustic leakage estimate, and at least one of: the internal audio signal, and the estimate of the at least one internal audio signal.

[0016] Generating at least one visualization based on the estimated audibility of external sounds so that the visualization provides an indication of the audibility of the external audio source, including displaying the estimated audibility of the external sounds using at least one of the following: a colour changing material; at least one light emitting diode; a display element; at least one liquid crystal display element; at least one organic light emitting diode display element; and at least one electrophoretic display element.

[0017] The device may include one of: a smart phone; a headset; a vehicle equipped with at least one external microphone; a helmet equipped with at least one external microphone; and personal protection equipment equipped with the at least one external microphone.

[0018] According to a second aspect, a method for controlling the sound audibility of an external audio signal within a device is provided, the method comprising: obtaining at least one input; determining whether the at least one input is provided by a user of the device or some other person; and controlling the sound audibility of the external audio signal to the user based on the at least one input and whether the at least one input is provided by the user of the device or the other person.

[0019] Controlling the sound audibility of the external audio signal to the user based on the at least one input and whether the at least one input is provided by the user of the device or the other person may include at least one of the following: switching between automatic noise control and transparency mode after determining a large control input from the user; switching from automatic noise control to full transparency mode after determining a down swipe from the user; switching from full transparency mode to automatic noise control after determining an upward swipe from the user; changing the internal volume based on a small control input from the user; switching between automatic noise control and transparency mode after determining any control input from the other person.

[0020] According to a third aspect, there is provided an apparatus for visualizing the sound audibility of an external audio signal, the apparatus comprising at least one processor and at least one memory, the memory storing instructions which, when executed by the at least one processor, cause the system to at least: obtain at least one external audio signal; obtain at least one of: an internal audio signal, and an estimate of at least one internal audio signal; estimate the external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of at least one internal audio signal; and generate at least one visualization based on the estimated external sound audibility, such that the visualization provides an indication of the audibility of an external audio source.

[0021] The device that is configured to perform acquisition of at least one external audio signal may be configured to perform acquisition of at least one external microphone audio signal, wherein the at least one external microphone is located on or acoustically coupled to an external surface of the device so that the at least one external microphone audio signal is configured to capture audio external to the device.

[0022] The device that is configured to perform acquisition of at least one internal audio signal may be configured to perform acquisition of at least one internal microphone audio signal, wherein the at least one internal microphone is located at or acoustically coupled to an internal surface of the device so that the at least one internal microphone audio signal is configured to capture audio inside the device.

[0023] The device being caused to perform obtaining an estimate of at least one internal audio signal may be caused to perform estimating at least one internal audio signal to be output by a transducer within the device, such that the estimate of the at least one internal audio signal is configured to assist in estimating the external sound audibility.

[0024] The device is caused to perform an estimation of external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of the at least one internal audio signal, and can be caused to perform: determining an acoustic leakage estimate based on the at least one external audio signal and a function defining a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; and generating the at least one external sound audibility based on subtracting the anti-noise audio signal from the acoustic leakage estimate.

[0025] The device is caused to perform an estimation of external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of at least one internal audio signal, and can be caused to perform: determining an acoustic leakage estimate based on the at least one external audio signal and a function defining a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; and generating the at least one external sound audibility based on subtracting the anti-noise audio signal from the acoustic leakage estimate, and at least one of: the internal audio signal, and the estimate of the at least one internal audio signal.

[0026] The device, which is caused to generate at least one visualization based on the estimated audibility of the external sound so that the visualization provides an indication of the audibility of the external audio source, may be caused to display the estimated audibility of the external sound using at least one of the following: a color-changing material; at least one light-emitting diode; a display element; at least one liquid crystal display element; at least one organic light-emitting diode display element; and at least one electrophoretic display element.

[0027] The device may include one of: a smartphone; a headset; a vehicle equipped with at least one external microphone; a helmet equipped with at least one external microphone; and personal protective equipment equipped with the at least one external microphone.

[0028] According to a fourth aspect, a device for controlling the sound audibility of an external audio signal is provided, the device comprising at least one processor and at least one memory, the memory storing instructions which, when executed by the at least one processor, cause the system to at least perform: obtaining at least one input; determining whether the at least one input is provided by a user of the device or by some other person; and controlling the sound audibility of the external audio signal for the user based on the at least one input and whether the at least one input is provided by the user of the device or by the other person.

[0029] The device, which is caused to control the sound audibility of the external audio signal to the user based on the at least one input and whether the at least one input is provided by the user of the device or the other person, may be caused to perform at least one of the following: switching between automatic noise control and transparency mode after determining a large control input from the user; switching from automatic noise control to full transparency mode after determining a downward swipe from the user; switching from full transparency mode to automatic noise control after determining an upward swipe from the user; changing the internal volume based on a small control input from the user; switching between automatic noise control and transparency mode after determining any control input from the other person.

[0030] According to a fifth aspect, there is provided an apparatus for visualizing the sound audibility of an external audio signal, the apparatus comprising a module configured to: obtain at least one external audio signal; obtain at least one of: an internal audio signal and an estimate of at least one internal audio signal; estimate the external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of at least one internal audio signal; and generate at least one visualization based on the estimated external sound audibility, such that the visualization provides an indication of the audibility of an external audio source.

[0031] The module configured to acquire at least one external audio signal may be configured to acquire at least one external microphone audio signal, wherein the at least one external microphone is located on or acoustically coupled to an external surface of the device such that the at least one external microphone audio signal is configured to capture audio external to the device.

[0032] The module configured to acquire at least one internal audio signal may be configured to acquire at least one internal microphone audio signal, wherein the at least one internal microphone is located on or acoustically coupled to an internal surface of the device such that the at least one internal microphone audio signal is configured to capture audio inside the device.

[0033] The module configured to obtain an estimate of at least one internal audio signal may be configured to estimate at least one internal audio signal to be output by a transducer within the device, such that the estimate of the at least one internal audio signal is configured to assist in estimating the external sound audibility.

[0034] The module configured to estimate external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of at least one internal audio signal, may be configured to: determine an acoustic leakage estimate based on the at least one external audio signal and a function defining a relationship between the at least one external audio signal and an effective listening signal of a user; generate an anti-noise audio signal based on the at least one external audio signal; and generate the at least one external sound audibility based on subtracting the anti-noise audio signal from the acoustic leakage estimate.

[0035] The module configured to estimate the external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal and the estimate of at least one internal audio signal, may be configured to: determine an acoustic leakage estimate based on the at least one external audio signal and a function defining a relationship between the at least one external audio signal and an effective listening signal of a user; generate an anti-noise audio signal based on the at least one external audio signal; and generate the at least one external sound audibility based on subtracting the anti-noise audio signal from the acoustic leakage estimate and at least one of: the internal audio signal, and the estimate of at least one internal audio signal.

[0036] The module configured to generate at least one visualization based on the estimated audibility of external sounds such that the visualization provides an indication of the audibility of an external audio source can be configured to display the estimated audibility of external sounds using at least one of: a color-changing material; at least one light-emitting diode; a display element; at least one liquid crystal display element; at least one organic light-emitting diode display element; and at least one electrophoretic display element.

[0037] The device may include one of: a smartphone; a headset; a vehicle equipped with at least one external microphone; a helmet equipped with at least one external microphone; and personal protective equipment equipped with the at least one external microphone.

[0038] According to a sixth aspect, a device for controlling the sound audibility of an external audio signal is provided, the device comprising a module configured to perform the following operations: obtaining at least one input; determining whether the at least one input is provided by a user of the device or by some other person; and controlling the sound audibility of the external audio signal for the user based on the at least one input and whether the at least one input is provided by the user of the device or by the other person.

[0039] The module configured to control the sound audibility of the external audio signal to the user based on the at least one input and whether the at least one input is provided by the user of the device or the other person can be configured to at least one of the following: switch between automatic noise control and transparency mode after determining a large control input from the user; switch from automatic noise control to full transparency mode after determining a down swipe from the user; switch from full transparency mode to automatic noise control after determining an upward swipe from the user; change the internal volume based on a small control input from the user; switch between automatic noise control and transparency mode after determining any control input from the other person.

[0040] According to a seventh aspect, a device for visualizing the sound audibility of an external audio signal is provided, the device comprising: an acquisition circuit configured to acquire at least one external audio signal; an acquisition circuit configured to acquire at least one of the following: an internal audio signal and an estimate of at least one internal audio signal; an estimation circuit configured to estimate the audibility of the external sound based at least in part on the at least one external audio signal and at least one of the following: the internal audio signal, and the estimate of at least one internal audio signal; and a generation circuit configured to generate at least one visualization based on the estimated external sound audibility such that the visualization provides an indication of the audibility of an external audio source.

[0041] According to an eighth aspect, a device for controlling the sound audibility of an external audio signal is provided, the device comprising: an acquisition circuit configured to acquire at least one input; a determination circuit configured to determine whether the at least one input is provided by a user of the device or by some other person; and a control circuit configured to control the sound audibility of the external audio signal for the user based on the at least one input and whether the at least one input is provided by the user of the device or by the other person.

[0042] According to a ninth aspect, there is provided a computer program comprising instructions or a computer-readable medium comprising instructions, the instructions being used to cause a device to visualize the sound audibility of an external audio signal, so that the device performs at least the following operations: obtaining at least one external audio signal; obtaining at least one of the following: an internal audio signal and an estimate of at least one internal audio signal; estimating the external sound audibility based at least in part on the at least one external audio signal and at least one of the following: the internal audio signal, and the estimate of at least one internal audio signal; and generating at least one visualization based on the estimated external sound audibility, so that the visualization provides an indication of the audibility of an external audio source.

[0043] According to the tenth aspect, there is provided a computer program [or a computer-readable medium] comprising instructions, wherein the instructions are used to cause an apparatus to control the sound audibility of an external audio signal, so that the apparatus performs at least the following operations: obtaining at least one input; determining whether the at least one input is provided by a user of the apparatus or by some other person; and controlling the sound audibility of the external audio signal to the user based on the at least one input and whether the at least one input is provided by the user of the apparatus or by the other person.

[0044] According to an eleventh aspect, a non-transitory computer-readable medium comprising program instructions is provided, wherein the program instructions are used to enable an apparatus for visualizing the sound audibility of an external audio signal, so that the apparatus performs at least the following operations: obtaining at least one external audio signal; obtaining at least one of the following: an internal audio signal, and an estimate of at least one internal audio signal; estimating the external sound audibility based at least in part on the at least one external audio signal and at least one of the following: the internal audio signal, and the estimate of at least one internal audio signal; and generating at least one visualization based on the estimated external sound audibility, so that the visualization provides an indication of the audibility of the external audio source.

[0045] According to the twelfth aspect, a non-transitory computer-readable medium comprising program instructions is provided, wherein the program instructions are used to enable an apparatus for controlling the sound audibility of a visualized external audio signal to perform at least the following operations: obtaining at least one input; determining whether the at least one input is provided by a user of the apparatus or by some other person; and controlling the sound audibility of the external audio signal to the user based on the at least one input and whether the at least one input is provided by the user of the apparatus or by the other person.

[0046] According to a thirteenth aspect, there is provided an apparatus for visualizing the sound audibility of an external audio signal, the apparatus comprising: a module for acquiring at least one external audio signal; a module for acquiring at least one of: an internal audio signal, and an estimate of at least one internal audio signal; a module for estimating the audibility of external sounds based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of at least one internal audio signal; and a module for generating at least one visualization based on the estimated external sound audibility such that the visualization provides an indication of the audibility of an external audio source.

[0047] According to the fourteenth aspect, a device for controlling the sound audibility of an external audio signal is provided, the device comprising: a module for obtaining at least one input; a module for determining whether the at least one input is provided by a user of the device or some other person; and a module for controlling the sound audibility of the external audio signal for the user based on the at least one input and whether the at least one input is provided by the user of the device or the other person.

[0048] According to a fifteenth aspect, there is provided a computer-readable medium comprising instructions for causing an apparatus for visualizing the sound audibility of an external audio signal to perform at least the following operations: obtaining at least one external audio signal; obtaining at least one of the following: an internal audio signal, and an estimate of at least one internal audio signal; estimating the external sound audibility based at least in part on the at least one external audio signal and at least one of the following: the internal audio signal and the estimate of at least one internal audio signal; and generating at least one visualization based on the estimated external sound audibility, such that the visualization provides an indication of the audibility of an external audio source.

[0049] According to the sixteenth aspect, a computer-readable medium comprising instructions is provided, wherein the instructions are used to cause a device for controlling the sound audibility of an external audio signal to perform at least the following operations: obtaining at least one input; determining whether the at least one input is provided by a user of the device or by some other person; and controlling the sound audibility of the external audio signal for the user based on the at least one input and whether the at least one input is provided by the user of the device or by the other person.

[0050] An apparatus comprises modules for performing the actions of the method as described above.

[0051] A device is configured to perform the actions of the method described above.

[0052] A computer program comprises instructions for causing a computer to execute the method as described above.

[0053] A computer program product stored on a medium can cause an apparatus to execute the method described herein.

[0054] An electronic device may include the apparatus described herein.

[0055] A chipset may include the device described herein.

[0056] The embodiments of the present application are intended to solve the problems associated with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] For a better understanding of the present application, reference will now be made by way of example to the accompanying drawings, in which:

[0058] Figure 1a and 1b An example device with light suitable for implementing certain embodiments is shown;

[0059] Figures 2a to 2e shows an example apparatus according to certain embodiments and light patterns showing different exterior and interior sound conditions;

[0060] Figure 3 An example apparatus suitable for determining an audibility estimate (estimated leakage) in an external microphone-only configuration is shown in accordance with certain embodiments;

[0061] Figure 4 According to some embodiments, suitable Figure 3 A flowchart of the operation of an example apparatus for determining an audibility estimate (estimated leakage) and providing a visualization of the estimate is shown;

[0062] Figure 5 An example apparatus for providing a visualization of audibility estimates (estimated leakage) in certain embodiments is shown;

[0063] Figure 6 Another example apparatus suitable for determining an audibility estimate (estimating leakage) in an external microphone-only configuration in accordance with certain embodiments is shown;

[0064] Figure 7 Yet another example apparatus suitable for determining an audibility estimate (estimated leakage) in an internal microphone configuration according to some embodiments is shown;

[0065] Figure 8 According to some embodiments, it is suitable for Figure 7 A flowchart of the operation of an example apparatus for determining an audibility estimate (estimated leakage) is shown;

[0066] Fig. 9 An example apparatus having a user interface input suitable for implementing some embodiments is shown; and

[0067] Fig.10 Example apparatus suitable for implementing some embodiments is schematically shown. DETAILED DESCRIPTION

[0068] The embodiments discussed herein are intended to assist in the visualization of audibility for a user, for example, in a vehicle, wearing a helmet or personal protective equipment (PPE), or wearing headphones, earbuds, or certain head-mounted audio transducer systems. In the following embodiments, the device is equipped with an active noise cancellation (ANC) system. However, it will be appreciated that in some embodiments, the device is equipped with a passive noise cancellation or noise blocking system, and the examples described herein will similarly apply.

[0069] As mentioned above, determining and indicating the user's attention or in other words the audibility of "external" or local audio source sounds is a research topic. There are several reasons for conducting this research.

[0070] The upcoming Immersive Voice and Audio Services (IVAS) standard and immersive voice applications are configured to provide immersive audio communications. This type of communication system is more immersive, which means that the user's cognitive abilities will be more immersed in the communication than before.

[0071] In addition to the above scenarios, for headphones, it is annoying to take off the headphones and put them back on. In addition, earbud headphones are placed in the ears and fixed due to friction, which rubs the ears and causes discomfort when frequently inserted and removed, especially when the memory foam begins to lose elasticity after a certain number of insertions.

[0072] Thus, even when the user is not actively listening to an internal source or listening to and / or talking to others, they typically wear headphones or leave earbuds in their ears. As mentioned above, headphones typically may be equipped with a pass-through mode that may be selected and present the external audio signal to the headphone user without any cancellation.

[0073] This may annoy others when they try to connect or get the headphone user's attention. For example, a person trying to have a conversation with a headphone user may consider it antisocial or rude to not remove the headphone because the speaker cannot be sure that the headphone user can hear them.

[0074] It is not an easy task to estimate whether a headphone user can hear an external sound source, or in other words, to determine an audibility estimate of an external audio signal. For example, a headphone without an inner microphone has no direct way to enable audibility estimation.

[0075] Furthermore, audibility is affected by many factors that exacerbate the problem of estimating what a headphone-wearing user can hear. For example, audibility estimation should incorporate factors such as internal sound levels, headphone fit that affects acoustic leakage, and headphone playback.

[0076] Furthermore, even when audibility can be estimated and visualized, the ability to change audibility (such as conventional methods for controlling ANC / transparency and internal sound levels) can be cumbersome because they typically require the use of at least two separate controls. As a result, the talker (or headset user) cannot easily change audibility as might be required in an emergency situation.

[0077] This problem may also arise in the scenarios described above. For example, rather than trying to get the attention of a user wearing a headset, the speaker may be trying to get the attention of a car driver (e.g., requesting identification when passing through a security checkpoint), a motorcyclist (e.g., instructing to remove a helmet), or a PPE wearer (e.g., to indicate to the person wearing PPE where their field of vision is restricted by the PPE).

[0078] Therefore, it is an object of the embodiments discussed in further detail to provide apparatus and methods for estimating the audibility, and for example the extent to which a headphone user hears external sounds relative to all sounds (external + internal) when wearing the headphones, and indicating this to others.

[0079] Furthermore, in some embodiments, a simple or single control interface is provided that can be configured to enable changes in audibility, for example to simultaneously control the ANC / transparency metric and the internal sound level.

[0080] These apparatus and methods can be used in immersive audio communications based on the IVAS standard and immersive speech. These embodiments are particularly suitable for this type of communication because they are immersive, which means that the user's cognitive ability will be more immersed in the communication than in a traditional audio communication call. Users in an immersive audio communication call may not always pay equal attention to their surroundings, so visualizing audibility and thus indicating an "attention level" based on an estimate of audibility based on external and internal audio signal components enables others to recognize whether they can be heard.

[0081] The concepts discussed in further detail in the embodiments below may be methods and devices that provide an estimate of the audibility of external sounds with respect to the first aspect and are capable of displaying the estimated audibility of external sounds in an appropriate manner. External sound audibility may, for example, be represented as the audibility of external sounds associated with a headset that does not include a built-in microphone (or in other words, only includes an external microphone). However, external sound audibility may more generally refer to the ability of a user to hear another person, where the parties are at least partially acoustically separated or disconnected. For example, a user may be in a vehicle, or wearing a helmet or other personal protective equipment (PPE). Thus, the ability of a user to hear a speaker while wearing a helmet (or in a vehicle) can be displayed to the speaker.

[0082] The following examples will focus on the concepts of estimation of external sound audibility as applied to when a user is wearing headphones, where the module for estimating the external sound audibility and the module for displaying the estimated external sound audibility are implemented within the headphones. However, it will be appreciated that the following may be applied to other scenarios, such as wearing a helmet or PPE or within a vehicle, where the modules for estimation and display are implemented within these devices or devices associated with these devices. For example, in the case where a user is wearing PPE, the PPE may be equipped with an external microphone and an internal transducer, but may also be connected (e.g. wirelessly) to a mobile device that provides a module for estimation and may also provide a module for displaying the estimation.

[0083] In such an embodiment, the apparatus is configured to use at least the active noise cancellation (ANC) signal of the headset at low frequencies (since this signal is already available) as a starting point for generating an estimate of the audibility of the external sound.

[0084] In some embodiments, estimates of the audibility of external sounds may be displayed or visualized to others to see whether the (headphone) user can (or is likely to) hear them.

[0085] Additionally, in some embodiments, an estimate of external sound audibility may be determined (to the headphone user) based on internal sound (from the built-in microphone), but the internal sound level is not directly visualized to maintain the privacy of the internal sound. However, in such embodiments, external sound audibility is estimated and a visualization of external sound availability is provided to others.

[0086] In some further embodiments, devices and methods are provided that are configured to control ANC / transparency and internal sound levels using a single or simple volume control button / slider. In some embodiments, the control mechanism is intended to keep the internal sound level at the lowest possible level by using ANC / transparency as much as possible to attenuate external sounds. In some further embodiments, instead of increasing the internal volume, the ANC function is increased (or placed on a higher setting). In these embodiments, the internal sound is controlled to sound louder because the external noise is attenuated by the ANC. In these examples, the louder the bass frequencies in the external sound, the more often the ANC / transparency control is used to replace the volume control, because the ANC / transparency control effect is most obvious when there is a lot of bass in the external sound. In this way, the user's hearing is protected and the user can better hear the external sound source.

[0087] In some embodiments, a simple or single (volume) control button / slider may be configured such that a large slide from the headphone user may cause the internal sound level to decrease / increase and control the ANC switch to switch between transparency mode / ANC mode, respectively. Additionally, in some examples, a small slide by the headphone user causes only the internal sound level to change.

[0088] Additionally, in some examples, if someone other than the headset user slides the volume control, the ANC is configured to (always) change to transparency mode. In this way, someone else can quickly get the headset user's attention and the other person's voice can be heard.

[0089] The visualization may be a light color, light brightness, light flashing pattern, or a more complex image-based or message-based visualization. The visualization may be represented by a single light (or light emitting diode) or multiple lights or arrays of lights (or light emitting diodes). In some embodiments, the visualization may be any suitable display technology, such as an LCD, OLED, or electrophoretic display. In some embodiments, the visualization may be achieved by a color-changing material, where the color of a product (e.g., a headset, a smart phone, a vehicle, etc.) may be partially or completely changed. For example, a PPE helmet may include at least one strip of color-changing material that is configured to change color based on whether the wearer is able to hear an external speaker.

[0090] Figure 1a and Figure 1b An example potential visualization that a headset could display is shown in FIG.

[0091] For example, Figure 1aA user 101 is shown wearing a pair of headphones 103, and another person 107 is speaking in a "loud" voice 109. The headphones 103 may include an LED (light emitting diode) or other light source, whose output is configured to indicate that the user 101 can hear the other person 107. First visualization 105 (which may be a light pattern, color, brightness, image, or message based on the example embodiment) indicates that the headset has "good" audibility to the other person 107. In this example, the visualization is a first color, for example but not shown, the visualization is a green light.

[0092] also, Figure 1b The same user 101 is shown with a headset 103 and another person 107 speaking in a "quiet" voice 110. The LED or other light source is configured to output a second visualization 106 that is configured to indicate that the user 101 cannot hear the other person 107. In other words, the second visualization 106 indicates that the audibility of the headset is "poor." In this example, the visualization is a second color, for example but not shown, a red light.

[0093] Although Figure 1a and Figure 1b The examples shown in show whether the user can hear another person (or external source speaker) or indicate good / poor audibility, but the visualization does not reflect internal and external sound conditions.

[0094] Further visualization examples of audibility based on internal and external sound conditions are shown in Figures 2a to 2e shown.

[0095] For example, Figure 2a A user 101 is shown wearing a pair of headphones 103 while another person 107 is speaking with a "loud" voice 109. In this example, the visualization or display module includes a series of LEDs or other light sources that output a first visualization that is configured to indicate "good" audibility based on internal and external sound conditions, and that the user 101 is able to hear the other person 107. In this example, the visualization is displayed by three LEDs, including a top, middle, and bottom LED. In this "good" audibility or sound condition example, the LEDs all display a first color, a green top LED 201, a green middle LED 203, and a green bottom LED 205. In this example and other examples, the color or arrangement selection of the LEDs can vary based on the application, as long as the visualization is consistent.

[0096] Figure 2bA situation is shown where another person 107 is speaking with a "quiet" voice 110 and is therefore an example of a "poor" audibility or sound condition. In this example, the visualization is configured to indicate that the user 101 cannot hear the speaker at a low volume level. In this example, the "poor" audibility or sound condition example has a visualization that uses LEDs in such a way that the top LED has a first color, green top LED 201, but the middle and bottom LEDs have a second color, red middle LED 213 and red bottom LED 215.

[0097] Figure 2c A situation is shown where another person 107 is speaking at a "sufficient" volume voice 209, and therefore in this example, there is no external noise or interfering audio sources or internal audio sources, and there is an "adequate" audibility or sound condition example. In this example, the visualization is configured to indicate that the user 101 is (just) able to hear the speaker. In this example, the "adequate" audibility or sound condition example has a visualization that uses LEDs in a manner where the top and middle LEDs have a first color, green top LED 201, green middle LED 203, but the bottom LED has a second color, red bottom LED 215.

[0098] Figure 2d Another situation is shown, where another person 107 is speaking at a "sufficient" volume 209, but there is an external audio source 221 that produces noise or interfering audio signals 222 that cause the audibility or sound condition example to be "poor". In this example, the visualization is configured to indicate that the user 101 cannot hear the speaker at a "sufficient" volume level due to the external audio source. In this example, the "poor" audibility or sound condition example has a visualization effect using LEDs in such a way that the top LED has a first color, i.e., a green top LED 201, and the middle and bottom LEDs have a second color, i.e., a red middle LED 213 and a red bottom LED 215.

[0099] Figure 2e Another situation is shown, where another person 107 is speaking 209 at a "sufficient" volume level, but there is also an internal audio source 231 (music input) that effectively degrades the sound conditions and results in a "poor" audibility or sound condition example. In this example, the visualization is configured to indicate that the user 101 is unable to hear the speaker at a "sufficient" volume level due to the internal audio source. In this example, the "poor" audibility or sound condition example has a visualization effect using LEDs in such a way that the top LED has a first color, i.e., a green top LED 201, and the middle and bottom LEDs have a second color, i.e., a red middle LED 213 and a red bottom LED 215.

[0100] about Figure 3 , shows a schematic diagram of an example device suitable for implementing certain embodiments. ANC-based headphones can be configured without a built-in microphone (a built-in microphone is a microphone between the speaker and the eardrum). This configuration can be implemented to save costs, and can also be implemented in order to reduce weight and / or packaging size. Therefore, cheaper and / or smaller headphones can be designed in this way, because they cannot be equipped with a large enough battery and / or computing power to use a large number of microphones.

[0101] A headset with a built-in microphone can be implemented, where observations and estimates can be made from the signals recorded by the built-in microphone.

[0102] exist Figure 3 In the example shown, the device and ANC features in the headset (without built-in microphone) include at least one external microphone 301 (which can be implemented as a microphone on the outer surface of the headset or with an acoustic opening on the outer surface of the headset). Figure 3 As shown, this embodiment may employ feed-forward ANC.

[0103] exist Figure 3 In the illustrated embodiment, at least one external microphone 301 generates captured audio 302 which is passed to a microphone gain (or microphone amplifier) ​​303 , the output 304 of which is passed to an acoustic leakage estimator 307 and a combiner 305 .

[0104] The combiner 305 is configured to output a combined anti-noise signal 306 having an anti-phase (and proportional to the noise level), which is passed to an ANC / transparency controller 309 and an audibility (leakage) estimator negative combiner 311 .

[0105] The acoustic leakage estimator 307 is configured to receive the signal 304 from the output of the microphone amplifier 303 and generate an acoustic leakage estimate 308 , which is also passed to the audibility (leakage) estimator negative combiner 311 .

[0106] The audibility (leakage) estimator negative combiner 311 receives the acoustic leakage estimate 308 and the combined anti-noise signal 306 and generates an audibility (leakage) estimate 312 .

[0107] The ANC / transparency controller 309 is also configured to receive the combined anti-noise signal 306 and output an ANC / transparency signal 310, which is passed to the combiner 313. The level of the anti-noise signal 306 may be controlled (by the user) in the ANC / transparency controller 309. For example, when ANC is required, the combined anti-noise signal 306 is fed to the speaker as such, and when transparency is required, the combined anti-noise signal 306 is reduced or completely zeroed.

[0108] The combiner 313 is also configured to receive the music (or internal audio source) audio signal 314 and output the combined audio signal to the transducer (or speaker) 315 of the headphone.

[0109] In other words, the device is configured to receive at least one externally mounted or coupled microphone audio signal as input and estimate the anti-noise signal (based on any suitable method) from the microphone audio signal therefrom. In the simplest case, the anti-noise signal 306 is just the inverted and low-pass filtered external microphone signal 304.

[0110] Typically, the ANC / transparency controller 309 can also be configured to feed the microphone signal directly into the speaker when transparency is required ( Figure 3 not shown).

[0111] ANC is usually only implemented at lower frequencies because headphones (passively) mechanically attenuate higher frequencies very well. Also, at lower frequencies, latency in the creation and playback of the anti-noise signal is not as important. At lower frequencies, smaller latency is acceptable because the phase changes slowly and a slightly delayed anti-noise signal can still function as a good noise-canceling audio signal.

[0112] Thus, in some embodiments, the acoustic leakage estimator 307 receives an outer or external microphone audio signal and applies a transfer function thereto. The transfer function is configured to mimic the attenuation of external sound transmitted to the user's ear by the mechanical structure of the headphone.

[0113] In some embodiments, the transfer function is a frequency dependent attenuation configured to attenuate the microphone signal close to 0 dB at lower frequencies (eg, below 50 Hz) and increasingly more attenuate at higher frequencies (eg, 10 dB at 5 kHz).

[0114] By subtracting the ANC anti-noise signal 306 from the acoustic leakage estimate 308 (as shown by the audibility estimator negative combiner 311), an effect that simulates applying an ANC effect to the sound that the user can hear can be formed. In some embodiments, if the transparency mode is used or enabled, the anti-noise signal will switch to a signal that amplifies the sound, but the same subtraction will be performed in this case.

[0115] about Figure 4 , showing Figure 3 An example flow chart of the operation of the illustrated apparatus.

[0116] Therefore, the first operation of receiving the external microphone audio signal is shown, such as Figure 4 As shown in step 401.

[0117] Then, the operation of estimating the ANC signal from the external microphone audio signal is shown, as Figure 4 As shown in step 403.

[0118] also, Figure 4 The operation of estimating acoustic leakage from the external microphone audio signal is also shown in FIG. Figure 4 As shown in step 405.

[0119] After the ANC and acoustic leakage audio signals are determined, an audibility estimate or sound condition estimate may be determined based on the difference between the ANC and acoustic leakage audio signals, such as Figure 4 As shown in step 407.

[0120] The audibility estimate or sound condition estimate can then be used as input to generate a visualization to indicate what sounds the user can hear, e.g. Figure 4 As shown in step 409.

[0121] about Figure 5 , is an example apparatus for generating a visualization from an audibility (leakage) estimate 312 .

[0122] In this example, the apparatus includes a smoother 501 configured to receive the audibility (leakage) estimate 312 and generate a smoothed version to pass to an analog-to-digital converter (ADC) and logic circuit 503. The smoothing may be any suitable smoothing, for example, low pass filtering of the estimate. In some embodiments, the smoothing is implemented on a band-by-band basis and the band values ​​are combined after weighting using a weighting function.

[0123] The ADC and logic circuit 503 is configured to receive the smoothed value and control at least one LED based on the average value.For example, the audibility (leakage) estimate may be used to indicate, for example to other users, whether the headphone user can hear them.

[0124] Thus, for example, the outputs 5040 to 504 by the ADC and logic circuits are shown. n The controlled or driven LED array 5050 to 505n. Therefore, the higher the average value, the more LEDs are powered (or switched from red to green).

[0125] The LED light can be used to indicate this because they can see from the light if the other person's voice causes a visible change in the LED light. In other words, when the other person speaks, the LED light changes, then the other person knows that the headset user can hear them. If the LED light does not change significantly, then they know that the headset user cannot hear them.

[0126] In some embodiments, the greater the audibility estimate, the more LED lights are illuminated. Additionally or alternatively, in some embodiments, as described above, the LED color can also be used to indicate whether the headset user can hear other people's voices (and typically green can indicate that the headset user can hear other people's voices).

[0127] In some embodiments, the LED or visualization and audibility estimation points to the audibility of speech. This can be achieved using any suitable method or device. For example, a bandpass filtered signal is used, where the bandpass filter is centered on the dominant speech frequency. In other words, the frequency band that determines the audibility estimate is typically 400Hz-4kHz, so the bandpass filter center frequency is about ~1kHz. In some embodiments, a voice activity detection (VAD) algorithm can be used to detect when speech is present, and then audibility estimates are made for these detected segments.

[0128] In some embodiments, a speech separation algorithm may be used to detect the amount of speech and determine the audibility estimate based solely on that determined amount.

[0129] In some embodiments, the audibility estimate (based on leakage) is further determined based on the internal audio signal (music audio signal 314). Figure 6 Shows something like Figure 3 , but the leakage audibility estimator 601 is configured to receive the output of the audibility (leakage) estimator negative combiner 311 or the leakage audio 610 and the music audio signal 314 to generate the audibility (leakage) estimate 612. The leakage audibility estimator 601 is configured to compare the leakage audio 610 with the internal audio signal 314 (music). In some embodiments, this comparison is a simple energy ratio ((leakage audio) / (internal audio)), or an energy ratio in a frequency band, or the leakage audio can be compared with a masking threshold calculated from the internal audio signal.

[0130] In some embodiments, if the leaked audio is below the masking threshold or, for example, the energy ratio is below -10 dB, the leakage audibility estimate is 0. The more the leaked audio is above the threshold or the greater the energy ratio, the greater the audibility estimate. In some embodiments, the estimate can be scaled and limited so that the estimate reaches a maximum value of 1 when the leaked audio is 30 dB above the masking threshold or the energy ratio is above 20 dB.

[0131] In some embodiments, Figure 7 Another example arrangement is shown in which a built-in microphone is used to generate an audio signal as a basis for audibility estimation.

[0132] exist Figure 7In the illustrated embodiment, at least one internal microphone 701 generates captured audio 702 , which is passed to a microphone gain (or microphone amplifier) ​​703 , the output 704 of which is passed to a negative combiner 705 .

[0133] In addition, the music audio signal 714 (or internal source) is passed to the music amplifier 724, which also outputs an amplified music audio signal 734 to the negative combiner 705.

[0134] The negative combiner 705 generates a combined anti-noise signal 706 having an inverted phase (and proportional to the noise level) which is passed to the combiner 713 and to the leakage audibility estimator 713 as the leakage estimate 712 .

[0135] The combiner 713 is configured to receive the music (or internal audio source) audio signal 714 and the combined anti-noise signal 706 and output the combined audio signal to the transducer (or speaker) 715 of the headphone.

[0136] The leakage audibility estimator 713 is configured to receive the music (or internal audio source) audio signal 714 and the combined anti-noise signal (as the leakage estimate 712) and output an audibility (leakage) estimate 722 for generating a visualization.

[0137] In some embodiments, the audibility (leakage) estimate may be used to control the LEDs in a manner similar to that described above.

[0138] In these embodiments, the energy variations of the internal sound that the headphone user is listening to are hidden from others. Therefore, this improves the privacy of the user of the headphone.

[0139] about Figure 8 , showing Figure 7 An example flow chart of the operation of the illustrated apparatus.

[0140] Thus, a first operation of receiving an (internal) microphone audio signal is shown, such as Figure 8 As shown in step 801.

[0141] Also shown is the operation of receiving an internal audio signal (eg, a music audio signal). Figure 8 As shown in step 803.

[0142] Then, the operation of estimating the ANC signal from the external microphone audio signal is shown, as Figure 8 As shown in step 805.

[0143] In addition, Figure 8 The operation of estimating acoustic leakage from the internal microphone audio signal is shown in Figure 8As shown in step 807.

[0144] After the ANC and acoustic leakage audio signals are determined, an audibility estimate or sound condition estimate may be determined based on the difference between the ANC and acoustic leakage audio signals, such as Figure 8 As shown in step 809.

[0145] The audibility estimate or sound condition estimate can then be used as input to generate a visualization to indicate what sounds the user can hear, e.g. Figure 8 As shown in step 811.

[0146] When the volume of a headset is set too high, the user of the headset may suffer hearing damage over time. The most common reason for increasing the volume of the music is that external noise sources or interfering external sources make it difficult to hear the internal audio signal (e.g., the music audio signal). Therefore, when the user presses / slides the sound level control, the hearing of the headset user can be protected by increasing the ANC instead of the internal sound (music) level. This is also more practical than using two different controls implemented in known headsets.

[0147] As always, when the user controls something, there must be tangible results, otherwise the user will think the headset or device is broken. Therefore, using ANC to replace a music volume control should only be used in noisy environments. In quiet environments, there is no noise for the ANC to eliminate, so any changes in ANC operation are mostly unnoticeable. In addition, for the reasons mentioned above, the effect of applying ANC is generally greatest at lower frequencies. At higher frequencies, the effect comes from transparency mode, where external sounds are fed to the headset transducer or device speakers using an external or external microphone.

[0148] Therefore, in some embodiments, controls that affect both ANC and transparency are used. When transparency mode is at a full setting, typically ANC is mostly off, and at higher frequencies, the attenuation effect of the headphone body (higher passive attenuation at higher frequencies) is compensated by feeding external sound to the speakers. When transparency is at a lower setting, ANC is used to attenuate low frequencies, and at higher frequencies, the device relies primarily on passive attenuation of the device body.

[0149] Thus, in some embodiments, the "volume" change control is configured to implement the following actions based on determining whether background noise is present.

[0150]

[0151] In some embodiments, the controls are configured to provide different combinations and steps. The goal of such embodiments is to reduce the volume of the music as much as possible without sacrificing the user experience, i.e., the audio changes noticeably each time the volume button is pressed, and in addition, to use a transparency mode at a lower volume setting so that the user can hear the surrounding environment.

[0152] For example, Fig. 9 As shown, suitable inputs may be used. In this example, user 901 wears a headset 903, which includes a touch pad 905, with which a hand / finger 921 of user 901 may interact.

[0153] However, in some embodiments, the user control (e.g., touchpad 905) is configured to provide additional UI controls for the single button / switch / touchpad that controls ANC / transparency and internal volume functions when another user (as shown by hand 931) attempts to interact with it.

[0154] Thus, in some embodiments, the user control is configured to detect or determine a large control input. The large control input may be, for example, a swipe. For example, if full transparency is not already being used, a downward swipe is configured to automatically change the ANC / transparency from ANC to full transparency. Similarly, a large control input or "upward" swipe may be configured to change the transparency to ANC.

[0155] These large controls can be places where the headset user expects a noticeable change in audio level, which often means the user wants to hear the surrounding environment or not. Large controls can be detected from significant touchpad or volume control input, such as a swipe of more than 50% of the total length of the touchpad or a volume control input held down for more than 1 second.

[0156] In addition to controlling the ANC / Transparency modes, the controls also change the internal volume.

[0157] In some embodiments, the device is configured to detect whether the control is given by the headset user or by another person. If the input is provided by the headset user, the effects discussed previously may be applied. In some embodiments, the device determines that the control is provided by another user, and then if the control input from the other person is a "down" input, the headset is configured to switch to a transparent or pass-through mode, and if the control input from the other person is an "up" input, the headset is configured to turn on ANC.

[0158] Additionally, the control can be configured to control the internal volume. Since other people are generally not interested in the internal volume of the headset user, but only care whether the headset user can hear them, this control makes sense when implemented.

[0159] In some embodiments, the device can be configured to distinguish between controls provided by the headset user and others using any suitable method. For example, in some embodiments where controls are implemented in a mobile phone, the phone camera can be used to identify whether the person providing controls is wearing a headset. In some embodiments, wireless positioning technology (such as BT LE) can be used to locate the direction of the headset and whether the direction is in front of the phone, and then provide controls on the phone for the headset user, otherwise for others.

[0160] In some embodiments, controller identification can be based on fingerprint identification. In addition, a voice control interface can be used to identify who has control commands. A user can be identified using known user voice recognition methods, and ANC control operates as described above based on the identified user.

[0161] In addition, in some embodiments, device B belonging to another person (user B) can be used by user B to control the audibility of the headset user (user A) wearing the headset (device A) to user B. If user B brings device B near device A or points device B at device A, device B can indicate that it has synchronized with device A by giving feedback. Then, user B can use device B to control device A ANC. Proximity can be detected in any suitable manner, for example, proximity detection can be based on GPS, a camera detecting device A, or other positioning methods. The pointing detection module can use a Bluetooth LE antenna array that detects the direction of other Bluetooth devices. Feedback can be light or visualization on a display similar to that described above, or some tactile or vibration feedback. The control can be a touch screen or other touch detection sensor, a voice control or button, or a virtual button or switch. The user B device can be a wearable device, such as a ring, a headset, a watch, or a mobile phone.

[0162] For example, user B points their mobile phone (device B) at user A who is wearing headphones (device A), user B's phone displays a color on the display that is similar to the light in device A, and user B slides his finger on the displayed color to control the ANC of device A, allowing user A to hear user B's voice.

[0163] Control can also be wireless from an external device. For example, a museum could control the headsets it provides to museum visitors in safety or security situations. For example, when a fire alarm or other emergency notification needs to be issued to visitors.

[0164] While the above examples feature switching between ANC / Transparency modes, it will be appreciated that typical implementations allow for seamless switching between the two modes, with multiple intermediate steps, or a stepless change between "full" ANC and "full" transparency.

[0165] about Fig.10 , the example electronic device can be used as any device part of the system described above. The device can be any suitable electronic device or device. For example, in some embodiments, the device 2000 is a mobile device, a user device, a tablet computer, a computer, an audio playback device, etc. The device can be configured to implement an encoder or a renderer or any functional block as described above.

[0166] In some embodiments, the device 2000 includes at least one processor or central processing unit 2007. The processor 2007 may be configured to execute various program codes, such as the methods described herein.

[0167] In some embodiments, the device 2000 includes a memory 2011. In some embodiments, at least one processor 2007 is coupled to the memory 2011. The memory 2011 can be any suitable storage module. In some embodiments, the memory 2011 includes a program code portion for storing program codes that can be implemented on the processor 2007. In addition, in some embodiments, the memory 2011 may also include a storage data portion for storing data (e.g., data that has been processed or is to be processed) according to the embodiments described herein. The implemented program code stored in the program code portion and the data stored in the storage data portion can be retrieved by the processor 2007 via the memory-processor coupling when needed.

[0168] In some embodiments, the device 2000 includes a user interface 2005. In some embodiments, the user interface 2005 can be coupled to a processor 2007. In some embodiments, the processor 2007 can control the operation of the user interface 2005 and receive input from the user interface 2005. In some embodiments, the user interface 2005 can enable a user to enter commands to the device 2000, for example, via a keypad. In some embodiments, the user interface 2005 can enable a user to obtain information from the device 2000. For example, the user interface 2005 can include a display configured to display information to a user from the device 2000. In some embodiments, the user interface 2005 can include a touch screen or touch interface that can enable information to be input to the device 2000 and display information to a user of the device 2000. In some embodiments, the user interface 2005 can be a user interface for communication.

[0169] In some embodiments, the device 2000 includes an input / output port 2009. In some embodiments, the input / output port 2009 includes a transceiver. In such embodiments, the transceiver can be coupled to the processor 2007 and configured to enable communication with other devices or electronic devices, for example, via a wireless communication network. In some embodiments, the transceiver or any suitable transceiver or transmitter and / or receiver module can be configured to communicate with other electronic devices or devices via wires or wired couplings.

[0170] The transceiver can communicate with another device via any suitable known communication protocol. For example, in some embodiments, the transceiver can use a suitable Universal Mobile Telecommunications System (UMTS) protocol, a wireless local area network (WLAN) protocol such as, for example, IEEE 802.X, a suitable short-range radio frequency communication protocol such as Bluetooth, or an infrared data communication path (IRDA).

[0171] Input / output port 2009 may be configured to receive signals.

[0172] In some embodiments, device 2000 may be embodied as at least part of a renderer.Input / output port 2009 may be coupled to a headset (which may be a head-tracking or non-tracking headset) or the like.

[0173] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0174] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device (e.g., in a processor entity), or by hardware, or by a combination of software and hardware. In addition, in this regard, it should be noted that any block of the logic flow as shown in the figure may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip or a memory block implemented in a processor, a magnetic medium such as a hard disk or a floppy disk, and an optical medium such as a DVD and its data variant CD.

[0175] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology (e.g., semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory). The data processor may be of any type suitable for the local technical environment and may include one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a gate level circuit, and a processor based on a multi-core processor architecture, as non-limiting examples.

[0176] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is essentially a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0177] Programs, such as those offered by Synopsys Inc. of Mountain View, Calif., and Cadence Design Inc. of San Jose, Calif., use well-established design rules along with a library of pre-stored design blocks to automatically route conductors and locate components on a semiconductor chip. Once the design of a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transferred to a semiconductor fabrication facility or "fab" for manufacturing.

[0178] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0179] (a) pure hardware circuit implementation (e.g. implementation in analog and / or digital circuits only) and

[0180] (b) a combination of hardware circuitry and software, such as (where applicable):

[0181] (i) a combination of analog and / or digital hardware circuitry and software / firmware, and

[0182] (ii) any portion of a hardware processor (including a digital signal processor), software and memory with software that works together to enable a device such as a mobile phone or server to perform various functions, and

[0183] I A hardware circuit and / or a processor (eg, a microprocessor or portion of a microprocessor) that requires software (eg, firmware) for operation, but the software may not be present when it is not needed for operation.

[0184] This definition of circuitry applies to all uses of the term in this application, including in any claims. As another example, as used in this application, the term circuitry also covers only an implementation of a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and where applicable to a particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0185] The term "non-transitory" as used herein is a limitation of the medium itself (ie, tangible, not a signal), not a limitation of data storage persistence (eg, RAM vs. ROM).

[0186] As used herein, “at least one of: ” and “at least one of ” and similar expressions, where a list of two or more elements is connected by “and” or “or”, mean at least any one element, or at least any two or more elements, or at least all elements.

[0187] The foregoing description provides a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations will become apparent to those skilled in the relevant art in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications to the teachings of the present invention will still fall within the scope of the present invention as defined in the appended claims.

Claims

1. A method for visualizing the sound audibility of an external audio signal, the method comprising: Acquiring at least one external audio signal; Get at least one of the following: Internal audio signal; as well as an estimate of at least one internal audio signal; estimating external sound audibility based at least in part on the at least one external audio signal and at least one of: the internal audio signal, and the estimate of at least one internal audio signal; as well as At least one visualization is generated based on the estimated external sound audibility such that the visualization provides an indication of the audibility of the external audio source.

2. The method according to claim 1, wherein: Acquiring the at least one external audio signal includes acquiring at least one external microphone audio signal, wherein the at least one external microphone is located on or acoustically coupled to an external surface of the device such that the at least one external microphone audio signal is configured to capture audio external to the device.

3. The method according to claim 2, wherein: Acquiring the at least one internal audio signal includes acquiring at least one internal microphone audio signal, wherein the at least one internal microphone is located on or acoustically coupled to an interior surface of the device such that the at least one internal microphone audio signal is configured to capture audio inside the device.

4. The method according to claim 2, wherein: Obtaining the estimate of the at least one internal audio signal comprises estimating at least one internal audio signal to be output by a transducer within the device, such that the estimate of the at least one internal audio signal is configured to assist in estimating the external sound audibility.

5. The method according to claim 1, wherein: Estimating the external sound audibility includes: determining an acoustic leakage estimate based on the at least one external audio signal and a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; and The at least one external sound audibility is generated based on subtracting the anti-noise frequency signal from the acoustic leakage estimate.

6. The method according to claim 1, wherein: Estimating the external sound audibility includes: determining an acoustic leakage estimate based on the at least one external audio signal and a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; and The at least one external sound audibility is generated based on subtracting the anti-noise audio signal from the acoustic leakage estimate and at least one of: the internal audio signal and the estimate of the at least one internal audio signal.

7. The method according to claim 1, wherein: Generating the at least one visualization based on the estimated external sound audibility such that the visualization provides an indication of the audibility of the external audio source includes displaying the estimated external sound audibility using at least one of: Color-changing materials; at least one light emitting diode; Display components; at least one liquid crystal display element; at least one organic light emitting diode display element; and At least one electrophoretic display element.

8. The method according to claim 2, wherein: The device comprises one of the following: Smartphone; Headphones; a vehicle equipped with the at least one external microphone; a helmet equipped with said at least one external microphone; and Personal protective equipment equipped with the at least one external microphone.

9. A method for controlling the audibility of external sounds of at least one external audio signal in an apparatus, the method comprising: Get at least one input; determining whether the at least one input was provided by a user of the device or by another person; as well as Based on the at least one input and whether the at least one input is provided by the user of the apparatus or the other person, the external sound audibility of the at least one external audio signal to the user is controlled.

10. The method according to claim 9, wherein: Controlling the audibility of the external sound to the user includes at least one of: switching between automatic noise control and a transparency mode upon determining a large control input from the user; switching from automatic noise control to full transparency mode after determining a downward swipe from the user; switching from a full transparency mode to automatic noise control after determining an upward swipe from the user; varying an internal volume based on a small control input from the user; as well as Switching between automatic noise control and transparency modes occurs upon determining any control input from the other person.

11. An apparatus for visualizing the audibility of sound of an external audio signal, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: Acquiring at least one external audio signal; Get at least one of the following: Internal audio signals; and an estimate of at least one internal audio signal; estimating external sound audibility based at least in part on the at least one external audio signal, and at least one of: the internal audio signal, and the estimate of at least one internal audio signal; as well as At least one visualization is generated based on the estimated external sound audibility such that the visualization provides an indication of the audibility of the external audio source.

12. The device according to claim 11, wherein Causing the device to acquire at least one external audio signal causes the device to acquire at least one external microphone audio signal, wherein the at least one external microphone is located on or acoustically coupled to an external surface of the device so that the at least one external microphone audio signal is configured to capture audio external to the device.

13. The device according to claim 12, wherein: Causing the device to acquire the at least one internal audio signal causes the device to acquire at least one internal microphone audio signal, wherein the at least one internal microphone is located at or acoustically coupled to an internal surface of the device so that the at least one internal microphone audio signal is configured to capture audio inside the device.

14. The device according to claim 12, wherein: Causing the device to obtain the estimate of the at least one internal audio signal causes the device to estimate at least one internal audio signal to be output by a transducer within the device, such that the estimate of the at least one internal audio signal is configured to assist in estimating the external sound audibility.

15. The device according to claim 11, wherein Causing the device to estimate the external sound audibility causes the device to: determining an acoustic leakage estimate based on the at least one external audio signal and a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; as well as The at least one external sound audibility is generated based on subtracting the anti-noise frequency signal from the acoustic leakage estimate.

16. The device according to claim 11, wherein Causing the device to estimate the external sound audibility causes the device to: determining an acoustic leakage estimate based on the at least one external audio signal and a relationship between the at least one external audio signal and an effective listening signal of a user; generating an anti-noise audio signal based on the at least one external audio signal; as well as The at least one external sound audibility is generated based on subtracting the anti-noise audio signal from the acoustic leakage estimate and at least one of: the internal audio signal and the estimate of the at least one internal audio signal.

17. The device according to claim 11, wherein: causing the apparatus to generate the at least one visualization based on the estimated external sound audibility such that the visualization provides an indication of the audibility of the external audio source is configured to display the estimated external sound audibility using at least one of: Color-changing materials; at least one light emitting diode; Display components; at least one liquid crystal display element; at least one organic light emitting diode display element; as well as At least one electrophoretic display element.

18. The device according to claim 12, wherein: The device comprises one of the following: Smartphone; Headphones; a vehicle equipped with the at least one external microphone; a helmet equipped with said at least one external microphone; and Personal protective equipment equipped with the at least one external microphone.

19. An apparatus for controlling the audibility of external sounds of at least one external audio signal, the apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: Get at least one input; determining whether the at least one input was provided by a user of the device or by another person; as well as Based on the at least one input and whether the at least one input is provided by the user of the apparatus or the other person, the external sound audibility of the at least one external audio signal to the user is controlled.

20. The device according to claim 19, wherein Causing the device to control the audibility of the external sound to the user causes the device to at least do one of the following: switching between automatic noise control and a transparency mode upon determining a large control input from the user; switching from automatic noise control to full transparency mode after determining a downward swipe from the user; switching from a full transparency mode to automatic noise control after determining an upward swipe from the user; varying an internal volume based on a small control input from the user; as well as Switching between automatic noise control and transparency modes occurs upon determining any control input from the other person.

Images