Audio optimization for noisy environments

Abstract

The present invention relates to a method for enhancing speech intelligibility of an audio stream (20) containing speech audio. The method is performed in real time by an audio device and includes the steps of detecting (40) ambient noise and estimating (40) an internal noise based on the ambient noise. An audio filter (50) is determined based on the estimated internal noise and the audio stream (20), and the audio filter (50) is applied to the audio stream (20) to provide a target audio stream (20'). The target audio stream (20') is output to one or more transducers (35) to generate an internal sound (37) of the audio device. An internal sound (37) of the audio device (30) is detected, and the determination of the audio filter (50) is further based on the detected internal sound (37). An audio device and the like are also presented.

Description

[Technical Field]

[0001] The present invention relates to sound optimization for audio, and more particularly to a method and device for providing sound optimization in noisy environments. [Background technology]

[0002] Portable electronic devices are used everywhere, by virtually everyone. For example, mobile phones are carried at all times and can be used to make calls or listen to audio. The audio listened to can be music, but podcasts and audiobooks are also becoming increasingly popular. As the use and portability of electronic devices for communication and entertainment increases, the risk that audio will be consumed in a noisy environment increases. With music, a noisy environment may only be a nuisance, but when listening to voice audio, a noisy environment can make speech intelligible due to the noise.

[0003] The speech intelligibility of speech audio depends on the signal-to-noise ratio, in this case the ratio between the speech audio and the noise. Historically, speech intelligibility has been improved by modifying the signal-to-noise ratio. A brute force approach is to amplify the audio signal so that it exceeds the noise, which, needless to say, can cause hearing damage to the person listening to the speech audio. Another approach is to reduce noise by configuring the headphones to attenuate external noise, or by utilizing active noise cancellation, if headphones are used. Noise attenuation depends on the acoustic design and the fitting of the headphones to the user. Active noise cancellation requires significant processing power, which increases material costs and energy consumption accordingly.

[0004] From the above, it can be seen that there is room for improvement. Summary of the Invention

[0005] It is an object of the present invention to provide a new type of voice optimization that improves over the prior art and eliminates or at least mitigates the above-mentioned drawbacks. More specifically, it is an object of the present invention to provide a method and an audio device that improves the intelligibility of speech or voiced audio in noisy environments. These objects are achieved by the techniques set out in the accompanying independent claims, with preferred embodiments defined in the associated dependent claims.

[0006] In a first aspect, a method for enhancing speech intelligibility of an audio stream containing speech audio is presented. The method is executed in real time by an audio device and includes detecting ambient noise, estimating internal noise based on the ambient noise, and determining an audio filter based on the estimated internal noise and the audio stream. The method further includes applying the audio filter to the audio stream to provide a target audio stream and generating an internal sound of the audio device by outputting the target audio stream to one or more transducers. Additionally, the method includes detecting an internal sound of the audio device, and determining the audio filter is further based on the detected internal sound and includes subtracting the estimated internal noise from the detected internal sound to provide a true audio stream. The method further includes updating the audio filter based on a difference between the target audio stream and the true audio stream.

[0007] In one variant, determining the audio filter further comprises comparing the estimated internal noise to one or more masking thresholds and updating the audio filter based on the comparison, which is advantageous as it provides an energy and computationally efficient way of determining whether audio is masked by noise.

[0008] In one variant, the one or more masking thresholds are calculated by performing a critical band analysis of the audio stream, which includes auditory masking by frequency spreading, which is advantageous as it increases the accuracy of the masking thresholds.

[0009] In one variant, the method further comprises filtering the audio stream to compensate for a hearing profile associated with a user of the audio device, which is beneficial as speech intelligibility is further enhanced and optimized for the user.

[0010] In one variant, the step of determining the audio filter is performed after filtering, such that the determination is based on the audio stream compensated for a hearing profile associated with the user of the audio device. This is advantageous because the same audio filter algorithm can be used regardless of the user, and the computational effort can be reduced since some compensation has already been applied via the hearing profile.

[0011] In one variant, determining the audio filter further comprises determining a playback phon based on the playback volume, and updating the audio filter is further based on an equal loudness contour associated with the determined phon, which is beneficial because speech intelligibility varies across volume, but not evenly across all frequencies, and compensates for this increased speech intelligibility regardless of playback volume.

[0012] In one variant, the step of determining the playback phone is further based on the internal sound, which is beneficial as it gives an accurate reading of the actual sound pressure level experienced by the user.

[0013] In one variant, the step of determining the audio filter further comprises a step of smoothing the gain of the audio filter in frequency by convolution with a frequency window function, which is beneficial as it removes undesired differences between adjacent frequency groups.

[0014] In one variant, the step of determining the audio filter further comprises averaging the gain of the audio filter using an exponentially weighted moving average comprising one or more weighting parameters, which is beneficial as it removes undesired differences between adjacent frequency groups.

[0015] In one variant, the step of determining the audio filter further comprises applying a configurable blend setting to select the degree to which the audio filter should be applied to the audio stream, which is beneficial as it makes the amount of improvement customizable, allowing the user to select the desired amount of compensation.

[0016] In one variant, the step of estimating the internal noise is implemented by one or more recurrent neural networks (RNNs). The use of RNNs is beneficial as it allows for an accurate and efficient way of estimating the internal noise.

[0017] In one variant, the ambient noise is detected by an external microphone operatively connected to the audio device, which advantageously provides an accurate measurement of the ambient noise.

[0018] In one variant, the ambient noise is limited to a maximum audio bandwidth of up to 10 kHz, preferably up to 8 kHz, which is beneficial as it further reduces the computational complexity of the method.

[0019] In one variant, the method further comprises applying active noise cancellation (ANC) to the audio stream after applying the audio filter, which is beneficial as it further reduces noise for internal sounds.

[0020] In a second aspect, an audio device is presented, comprising one or more transducers, at least one internal microphone positioned to detect internal sounds in a user's ear cavity, and a processing module operatively connected to the internal microphone, the one or more transducers, and the external microphone, the processing module configured to perform the method of the present invention.

[0021] In one variant, an external microphone is included in the audio device, which is advantageous as it makes data from the microphone easily available to the processing module.

[0022] In a third aspect, an audio system for enhancing speech intelligibility in real time is presented, the system comprising a portable electronic device operatively connected to an audio device and configured to communicate an audio stream including speech audio to the audio device, the audio device being an audio device according to the present invention.

[0023] In one variant, the ambient noise is sensed by an external microphone included within the electronic device, and the electronic device is further configured to transmit the ambient noise sensed by the external microphone to the audio device. This is beneficial because additional noise data can be provided by the external microphone of the audio device. Alternatively or additionally, the audio device can be configured without an external microphone, thus reducing the cost of the audio device.

[0024] In a fourth aspect, a computer program product is presented, the computer program product being configured, when executed by a processing module, to cause the processing module to perform the method of the present invention. [Brief explanation of the drawings]

[0025] In describing the embodiments of the present invention below, reference is made to the accompanying diagrammatic drawings that show, by way of example only, how the concepts of the present invention may be implemented.

[0026] [Figure 1a] 1 is a schematic diagram of an audio system according to an embodiment of the present invention; [Figure 1b] 1 is a schematic diagram of an audio system according to an embodiment of the present invention; [Figure 2] 2 is a schematic diagram of an audio filter and its signals according to an embodiment of the present invention; FIG. [Figure 3] 1 is a block diagram of a method for enhancing speech intelligibility of an audio stream according to an embodiment of the present invention; [Figure 4] 1 is an exemplary plot of speech optimization according to an embodiment of the present invention. [Figure 5] 1 is an exemplary plot of equal loudness contours according to an embodiment of the present invention. [Figure 6] 1 is an exemplary plot of speech optimization according to an embodiment of the present invention. [Figure 7] FIG. 2 is a block diagram of an audio filter audio decision according to an embodiment of the present invention. [Figure 8] 1 is a computer program product according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] Specific embodiments will now be described more fully with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of the invention as defined by the appended claims to those skilled in the art.

[0028] The term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically. Two or more items that are "coupled" may be integral with one another. The term "a" is defined as one or more, unless the present disclosure expressly requires otherwise. The terms "substantially," "approximately," and "about" are defined as most, but not necessarily the entirety, of what is specified, as understood by one of ordinary skill in the art. "Comprises" (also "comprises" in any form, such as "comprises" and "comprising"), "has" (also "has" in any form, such as "has" and "having"), "includes" (also "includes" in any form, such as "includes" and "comprising"), and "contains" (also "contains" in any form, such as "contains" and "containing") are open-ended linking verbs. Consequently, a method that "comprises," "has," "includes," or "contains" one or more steps is not limited to having, but only having, those one or more steps.

[0029] FIG. 1a is a simplified diagram of a real-time audio system 1 for improving speech intelligibility of an audio source signal 20 in the form of an audio stream 20. A corresponding audio system 1 is also depicted in schematic form in FIG. 1b, providing an alternative illustration of the location of components and features. The audio stream 20 comprises speech audio and may be any suitable form of streamed audio. The speech audio may be any speech audio in digital form, such as a podcast, an audiobook, or a voice communication. The audio stream 20 is typically transmitted from an electronic device 10 to an audio device 30. Transmission is preferably wireless via a suitable wireless interface, such as Bluetooth Low Energy (BLE), but may also be wired. The electronic device 10 may be any suitable electronic device 10, but is preferably a portable electronic device 10, such as a mobile terminal 10. The audio device 30 is preferably a pair of headphones 30 and may be of any suitable design, e.g., supra-aural, circum-aural, or intra-aural, although any audio device 30 operable with the present invention is suitable, as will be readily understood by those skilled in the art. The audio device 30 is adapted to receive the audio stream 20 from the mobile terminal 10, which means that if the audio stream 20 is communicated via BLE, the audio device comprises one or more BLE chipsets. In addition to this, the audio device 30 comprises one or more transducers 35, preferably two transducers 35, and the necessary hardware and software modules to convert the received audio stream 20 into an analog signal that can be output by the one or more transducers 35. The audio stream 20 is associated with a playback volume 15 set by any device 10, 30 of the audio system 1, which indicates the level at which the audio stream 20 should be played. The hardware of the audio device 30 includes at least one processing module 32.The processing module 32 may comprise a suitable MCU or multiple MCUs, one or more DSPs, one or more memories, and may further comprise the necessary hardware for communicating across the interface, for example a BLE transceiver.

[0030] The transducers 35 of the audio device 30 are configured to generate sound that is directed into the ear cavity of a user of the audio device 30. The audio device 30 includes one or more internal microphones 36 positioned to measure the sound generated by the transducers 35. The sound is preferably measured in the user's ear cavity when the audio device 30 is used by the user. Preferably, one internal microphone 36 is provided to measure the sound generated by each transducer 35.

[0031] The audio system 1 further comprises one or more external microphones 5. The external microphones 5 may be any suitable microphones 5 external to the audio device 30 and operably connected to the processing module 32 of the audio device 30. The external microphones 5 may be included in the audio device 30, for example, if the audio device 30 is a headset, and the external microphones may be positioned to detect the voice of a user of the audio device 30. Alternatively or additionally, the external microphones 5 may be included in the portable electronic device 10, for example, if the portable electronic device 10 is a mobile terminal 10.

[0032] Referring now to FIG. 2 , a conceptual overview of the present invention is provided. The present invention is preferably implemented on the audio device 30 presented herein. The speech intelligibility of the audio stream 20 is enhanced by an audio filter 50. This audio filter 50, applied to the audio stream 20, generates a target audio stream 20′ that is output to one or more transducers 35. The target audio stream 20′ is filtered to enhance speech intelligibility relative to the audio stream 20. Further details regarding the internal operation of the audio filter 50 are provided throughout this disclosure. The audio filter 50 is determined based on at least ambient noise 40 and internal sounds 37. The ambient noise 40 may be detected by one or more external microphones 5 operatively connected to the processing module 32 of the audio device 30. The internal sounds 37 are sounds in the ear cavity of a user of the audio device 30, or an estimate thereof. The internal sounds 37 are measured by an internal microphone 36 associated with the one or more transducers 35. Preferably, audio device 30 includes at least one audio filter 50 associated with each of the transducers 35 of audio device 30 .

[0033] Referring to FIG. 3 , a method 100 for enhancing speech intelligibility of an audio stream 20 comprising speech audio is outlined. Note that method 100 may be described as enhancing speech intelligibility by processing an audio stream 20 comprising speech audio. Method 100 may be performed by an audio device 30 presented herein. Each step of method 100 is described in further detail elsewhere in this disclosure. Method 100 includes a step 150 of outputting a target audio stream 20′ to a user of audio device 30. That is, an audio filter 50 is applied 140 to audio stream 20 to provide a target audio stream 20′ that is provided to a transducer 35 to generate an internal sound 37. The internal sound 37 is typically detected 160 by an internal microphone 36 detecting 110 ambient noise 40. This is typically accomplished by one or more external microphones 5 measuring ambient noise and providing this to a processing module 32 of the audio device 30. The ambient noise 40 is used to estimate 120 the internal noise 40', which is the noise perceived by the user's eardrum. The audio stream 20, the estimated internal noise 40' and the detected internal sound 37 are used to determine a speech filter 50.

[0034] In some embodiments of the method 100, the hearing profile HL(f k ) which is advantageous as the user's hearing impairments and / or disabilities are compensated for in addition to the speech intelligibility of the method. Preferably, the hearing profile HL(f k) compensation is applied to the audio source signal 20 prior to the step 130 of applying or determining the audio filter 50, so that the audio filter 50 is determined based on the audio stream 20 compensated for the user's hearing disabilities and / or impairments. This is beneficial because it effectively removes differences between users and the same method 130 for determining the audio filter 50 can be used for all users. Furthermore, since the compensation for hearing ability only affects the audio stream 20, it directly improves speech intelligibility in most cases. In addition to this, the hearing profile compensation can be taken into account in determining the audio filter 50, and no pre-processing is required until the audio filter 50 is applied 140. The hearing profile HL(f k ) and the ambient noise 40 is determined by the user's hearing profile HL(f k ) can be important because, without taking into account the speech intelligibility enhancement process results may have a reduced impact on the hearing ability of some users.

[0035] It should be mentioned that in audio systems 1 or audio devices 30 that utilize noise cancellation technology, such as active noise cancellation (ANC), the noise cancellation is preferably applied to the audio stream 20 after the audio filter 50. This is beneficial because noise cancellation reduces the noise level, but may also distort the audio signal, the degree of distortion depending on the configuration of the noise cancellation and the tuning and / or calibration of the noise cancellation technology.

[0036] The target audio stream 20' may be distorted or otherwise adversely affected by, for example, digital-to-analog converters, transducer operation, and the position of the audio device 30 on the user. It is therefore beneficial to detect the internal sounds 37 presented to the user's eardrums after noise has been removed by noise cancellation, compare the true audio stream 37', i.e., the internal sounds 37 after the internal noise 40' has been removed, with the target audio stream 20', and act to minimize the difference.

[0037] As already indicated, the audio system 1 is an audio system 1 with real-time constraints. The audio stream 20 is received as digital samples, either sample by sample or frame by frame. The collection of samples in a frame can be done elsewhere or as part of the system 1, for example by the electronic device 10. The audio stream 20 is received at a sample rate F s (samples / s) which are formed into an audio signal frame with frame (time) index 1. Audio stream 20 may be mono or stereo.

[0038] The audio filter 50 is preferably based on psychoacoustic masking and includes a speech intelligibility index or equivalent, such as an intelligibility index, speech transmission index, or short-term objective intelligibility, and a psychoacoustic model derived from the theory of tonal masking of noise. The frequency gain of the audio filter 50 is calculated such that the internal noise 40' is masked by the target audio stream 20', which is described in more detail in other sections of this disclosure.

[0039] The estimated internal noise 40' can be provided in several different ways. The input to the estimation 120 of the internal noise 40' is the ambient noise 40 detected by at least one of the external microphones 5 110. The ambient noise 40 is provided by the external microphone as a microphone signal, preferably represented in a frame-by-frame configuration substantially equal to that of the audio stream 20. The microphone signal may also be a stereo signal; such a signal is typically referred to as a dual-microphone signal. A dual-microphone signal includes two independent microphone signals formatted as a single stereo signal. As mentioned above, there may be multiple external microphones 5 in the system 1, and the step 120 of estimating the internal noise 40' may include, for example, determining to use only one microphone signal among all the microphone signals provided by the external microphones 5. The decision regarding which external microphone to use may be based, for example, on the highest signal level, proximity to the transducer, etc. All external microphones 5 may be processed separately to obtain the ambient noise 40 from each processed external microphone 5. The external microphones 5 may be processed to obtain a stereo signal and further to obtain the direction of the ambient noise 40 so that each transducer 35 may be associated with a different ambient noise 40.

[0040] As one skilled in the art will understand after digesting the teachings herein, a trade-off may be necessary between utilizing multiple external microphones 5 to detect 110 the ambient noise 40 based on resource management and optimization of available processing power in the real-time audio system 1. This trade-off may depend on the application. If the audio device 30 is, for example, a pair of headphones 30, and sufficient processing power is present, two separate estimates 120 of the internal noise 40' may be realized, one for the user's left ear and one for the user's right ear. However, if, for example, processing power is insufficient or there are stringent requirements regarding current consumption, a reasonable assumption is that the ambient noise 40 is substantially equal for the left and right ears, and the same internal noise 40' may be utilized for both ears.

[0041] The external microphone 5 may be sampled at a different sampling rate compared to the audio signal. Considering that the important frequency range for voice communication is up to 8 kHz, the external microphone 5 is band-limited to a maximum bandwidth of 10 kHz, or preferably 8 kHz. A lower bandwidth reduces processing load, memory load, and current consumption. The maximum bandwidth may be further reduced to further reduce processing load, memory load, and current consumption, but the maximum bandwidth must be traded off with the ambient noise 40 requirements.

[0042] As one skilled in the art will understand after reading this disclosure, it should be noted that the external microphone 5 generates a signal that includes both ambient noise 40 and additional sound sources. Since only ambient noise 40 is relevant, this means that, for example, echoes arising from the transducer and near-end talk from listeners engaged in a conversation can be beneficially filtered out from the signal generated by the external microphone 5. This is beneficial because it reduces the risk that additional sound sources will be misclassified as ambient noise 40.

[0043] One solution to avoid misclassifying additional sounds as ambient noise is to use one or more noise estimation techniques, for example, higher-order statistics, cepstral analysis, autoregressive modeling, or non-parametric methods such as Welch spectral and minimum variance methods. Typically, in a minimum-effort implementation, the method 100 may stop detecting 110 the ambient noise 40 and / or estimating 120 the internal noise 40' if an additional sound source is detected by the external microphone 5. The distinction between background noise and voice sources may be resolved using, for example, a voice activity detector (VAD).

[0044] In one embodiment of the present invention, the internal noise 40′ is estimated 120 by a recurrent neural network (RNN). This is described in more detail in other sections of this disclosure, but one advantage is that the implementation and configuration complexity of, for example, estimating 120 the internal noise 120, detecting additional sounds, voice detection, etc., is traded for the training and operation of an RNN, which is well described in machine learning theory.

[0045] Regardless of how the internal noise 40' is estimated 120, the representation of the internal noise 40' includes, for each auditory filter band or critical band b only, a mean energy value E of the internal noise 40'. v (b), b = 1, …, N B The concept of critical band b is explained in the following section.

[0046] As mentioned above, the frequency gain of the audio filter 50 is calculated such that the internal noise 40' is masked by the target audio stream 20'. To achieve this, the audio stream 20 is filtered, either in frequency or as a critical band b representation. This can be achieved by dividing the audio stream 20 into subframes, allowing for up to 50% overlap with the previous subframe. The subframes may be windowed using an appropriate window function, e.g., a Hamming window, a Hanning window, a triangular window, etc. The power spectrum P x (k)=│X(k)│ 2 is calculated using the sub-framed time domain data and a Fast Fourier Transform (FFT) implementation, where k is the frequency bin index. The resolution of the frequency transform is preferably 1 / 2 times the sample rate F s and the subframe size. Typically, a trade-off between resolution and resource demand is required.

[0047] All quantities described in the frequency domain are expressed in sound pressure level (SPL), and P x(k) / N is the power spectral density in SPL per frequency bin index k, referenced to a free-field reference point located in the listener's ear cavity. The conversion from digital signal to sound pressure level is performed by appropriate frequency-dependent scaling, one scaling frequency function per microphone 5, 36, and one scaling frequency function per transducer 35. The scaling functions may be predetermined and / or configurable and are preferably stored in a memory operatively coupled to the processing module 32. The scaling functions can be considered a calibration step performed once during the design or configuration of the audio device 30. Typically, but not exclusively, the scaling functions for the microphones 5, 36 consist of one scale value per frequency bin index k and can be estimated from the microphone frequency response. The scaling frequency function for the transducer 35 corresponds to the transducer frequency response, including scaling by the distance to the reference point, typically the listener's ear. Illustratively, for a pair of headphones 30, the scaling frequency function for the audio stream 20 is based on the frequency response of the transducer 35 referenced to the ear reference point ERP.

[0048] Cochlear model supports 20 audio streams B The number of critical bands is N. B can be set according to the desired resolution in the frequency domain, so as to directly control the granularity with which the audio stream 20 can be adjusted by the audio filter 50. As one skilled in the art will appreciate, there is a trade-off between frequency resolution and resource demands, with increasing resolution requiring a larger cochlear model and therefore both higher computational effort and more complex implementation. The inventors behind the present disclosure have chosen to use 20 frequency bands N BWe found that σ = 20 is a reasonable choice in terms of frequency resolution and computational complexity. Without loss of generality, the division into critical bands b may be performed using equivalent rectangular bandwidth ERB, scale, and gammatone filter banks. Other scales and filter types may be utilized to accurately provide a cochlear model. For a general signal, for each critical band b, the power spectrum is used to calculate the average energy

number

number

[0049] In the following, the psychoacoustic properties of masking are explained in some more detail, in particular how a first signal masks a second signal so that the second signal is not perceived. For purposes of explanation, it is assumed that the audio stream 20 is approximated by a tonal signal, but that the ambient noise 40 is a broadband noise, and the theory of tonal masking noise applies.

[0050] The masking threshold T(b) associated with critical band b is calculated based on a critical band analysis of the audio stream 20, which may include frequency masking, joint frequency masking due to spreading, and time masking due to gain smoothing.

[0051] Critical band analysis involves the generation of a critical band representation of the audio stream 20.

number

number

[0052] The simultaneous frequency masking due to spreading described above can be described by a spreading function SF that models the frequency spreading. The spreading function SF is given by

number

number

number

number

number

number

number

number

[0053] The audio system 1 is preferably configured to calculate at least one masking threshold T for use in sound optimization, i.e., in improving speech intelligibility. Depending on the application, e.g., headphones 30, etc., the masking thresholds calculated by the audio system 1 may be

number

number

[0054] Masking Threshold

number

number

number

[0055] Voice optimization in this context is a vector

number

[0056]

number

[0057] This disclosure describes the model (matrix

number

number

[0058] As mentioned above, audio stream 20

number

number

[0059] In contrast to ANC or noise reduction techniques where noise is measured by a combination of external microphone 5 and internal microphone 36, in the present invention, internal noise 40'

number

number

number

number

number

[0060] In a non-limiting example, amplification may be applied to the audio stream 20 to reduce the resulting masking threshold

number

number

number

[0061] Desired Objective Function

number

number

number

number

[0062] W is a diagonal weighting matrix, where the main diagonal in this example is populated with frequency band weightings given by the Speech Intelligibility Index (Note 3). The optimization explicitly exploits frequency spreading in the process, where frequencies should be amplified or attenuated using importance weighting across frequencies.

[0063] In another non-limiting example, the weighting matrix W may be populated based on the error between the target audio stream 20' provided to the transducer 35 and the corresponding detected internal sound 37 provided by the internal microphone 36 after the internal noise 40' has been removed, i.e., after the noise 120 of the estimated internal noise 40' has been subtracted from the signal provided by the internal microphone 36. In this case, the appropriate weighting matrix W may be based on the error in the frequency domain, preferably the auditory band domain, and even more preferably, the appropriate weight values ​​may be in the range of [0-1], and most preferably normalized to a root-mean-square value of 1, e.g.,

number

number

[0064] Optimal Gain

number

number

[0065] H g Note that (k) is updated once for each new frame l of the audio stream 20 and the internal noise 40'. Typically, audio frames are relatively short, and the number of samples in an audio frame may be less than the equivalent of 100 ms. On the other hand, the human ear has an integration time of 100-300 ms. In addition, when applying adaptive frequency adjustments, the listener must experience a stable tonal balance in the audio stream 20; failure to achieve this may cause discomfort to the user. Another aspect is frequency variation in the voice-optimized signal, i.e., the target audio stream 20'. If adjacent frequency bands are adjusted to too different degrees, discomfort may occur. All of these characteristics are subjective and will be known to those skilled in the art after reading this disclosure.

[0066] The inventors behind the present disclosure have recognized that the gain of the audio filter 50 can be manipulated to mitigate the above-mentioned subjective effects. In one embodiment, the gain of the audio filter 50 is smoothed in frequency by convolution with a frequency window function, e.g., a triangular window or the like, to ensure that isolated frequency bins are neither too amplified nor too attenuated compared to neighboring frequency bins, i.e., to limit the gain variation between frequency bins. In one embodiment, the window can be set to a typical value of [0.15, 0.7, 0.15], i.e., after convolution, the resulting gain in each frequency band consists of a 15 / 15 percent ratio of the neighboring band and a 70 percent ratio of the current band. Typically, since each critical band is more independent from neighboring bands the further apart they are in frequency, it may be inappropriate to include more than three to five critical bands in such a convolution operation. In another additional or alternative embodiment, the gain of the audio filter 50 is adjusted by a weighting parameter T i are averaged using an exponentially weighted moving average with the weighting parameter T i may be, for example, user selectable, or may be, for example, the integration time of the human ear, i.e., T i = 0.3, which will also effectively slow down the update rate and thus allow the user to tune their hearing to the frequency coloration.

[0067] As will be apparent to those skilled in the art after studying the method 100 according to this disclosure, the method 100 relates to processing (filtering, modifying) the audio stream 20 to enhance its speech intelligibility. In its most general form, the method 100 does not involve adding the inverse of the ambient noise 40 or estimated internal noise 40'. Rather, the method 100 modifies (filters, processes) the audio stream 20 with an audio filter 50. The audio filter 50 is a filter adapted to enhance speech intelligibility and is not interested in removing the ambient noise 40 or the estimated internal noise 40', but rather in adapting the audio stream 20 to enhance the intelligibility of the speech contained in the audio stream 20 when heard in a noisy environment. Filtering the speech audio in the audio stream 20 to enhance its intelligibility alters the frequency content of the speech audio. In other words, the voice of the person who spoke the speech audio may appear strange or distorted after being subjected to the audio filter, but the intelligibility of the speech will be enhanced.

[0068] It should be emphasized that the gain adjustment of the audio filter 50 will be distributed across all frequencies of the source signal bandwidth due to simultaneous masking and energy constraints. Therefore, at some frequencies where noise is masked, the audio source signal may be attenuated, and vice versa. This phenomenon is illustrated in FIG. 4, which shows an example of an audio stream 20 as a solid line, the internal noise 40' as a dense dashed line, the masking threshold T as a dashed line, and the target audio stream 20' as a dotted line. The optimization visualized in FIG. 4 is valid for one frame. The target audio stream 20' has energy redistributed to mask the internal noise 40' in all frequency bands. As can be seen in FIG. 4, when comparing the target audio stream 20' with the audio stream 20, the target audio stream 20' is amplified at some frequencies and attenuated at other frequencies by the audio filter 50 in accordance with the teachings herein. Note that if the internal noise 40' is below the masking threshold T for all frequencies, speech enhancement due to background noise is not necessary.

[0069] Where applicable, hearing impairment compensation may be described as, but is not limited to, a filter in either the time domain or the frequency domain that counteracts or reduces the user's hearing impairment. The hearing impairment is determined by the hearing profile HL(f k ) and the frequency function HL(f k ), f k denotes a set of discrete frequencies (typically a set of 5-7 frequencies is used) in units of dB HL, the hearing level per frequency. The hearing profile HL (f k ) is equal to or comparable to an audiogram, which is the result of a hearing test in which a tone audiogram is performed, but is not limited to this. No impairment corresponds to 0 dB HL, and increasing values, i.e., values ​​greater than 0, indicate hearing impairment or hearing loss. The creation of compensation to reduce hearing impairment is described below. In one embodiment, hearing impairment compensation is calculated using a frequency function H HI (k), i.e., the user's hearing profile HL(f k) and applied 105 to the audio stream 20 before speech optimization. Alternatively, the audio stream 20 can be included in speech optimization 130 without pre-processing. As previously shown, H HI (k) is grouped into critical band representations and frequency bin scaling is

number

[0070] In one embodiment, the user may select, for example via a mix setting, the degree m to which audio compensation should be applied to mask the ambient noise 40. For each frequency bin and m=[0,...,1],

number

number

[0071] User's hearing profile HL(f k), the present invention may optionally be combined with a loudness-dependent compensation VDC. As explained, the audio stream 20 includes a frequency spectrum, and this frequency spectrum of the audio signal is perceived differently at different playback sound pressure levels. This can be seen when comparing equal loudness curves 300 (see FIG. 5) at different SPLs. By compensating for the differences between the different equal loudness curves 300, the audio stream 20 is more likely to be perceived as intended, i.e., as perceived when mastered with a master for the source of pre-recorded data such as a podcast or audiobook.

[0072] A number of equal loudness curves 300 are shown in FIG. 5. The equal loudness curves 300 are a measure of SPL, typically expressed in decibels as dB sound pressure level (SPL). The sound pressure level is provided across a frequency spectrum, typically the human hearing spectrum. Each point within the equal loudness curves 300 represents a sound pressure level at which a listener perceives a certain loudness when compared to a pure tone presented at 1 kHz using a certain phon. A phon is defined in DIN 45631 and ISO 532 and is defined herein to mean that a 1 kHz tone with 50 dB SPL has a loudness of 50 phons, and all other frequencies perceived at the same loudness as this signal also have a loudness of 50 phons.

[0073] In FIG. 5, equal loudness curves 300 are plotted for six different phons, i.e., there are six plots on the graph in FIG. 5. In FIG. 5, equal loudness curves 300 are presented for 0, 20, 40, 60, 80, and 100 phons. The loudness curves 300 in FIG. 5 are ISO 226:213 loudness curves 300. These curves 300 are compared and corrected with the Fletcher-Munson curve provided by Harvey Fletcher and Wilden A. Munson in 1933. The Fletcher-Munson curve is usually referred to as the first explanation of the concept of equal loudness. If sound pressure level does not affect how the frequency spectrum of an audio signal is perceived by a listener, the offset between the curves 300 in FIG. 5 is the same across all frequencies.

[0074] In addition to or instead of the hearing profile filtering 105, the method 100 may further include a step 136 of determining a playback phone based on the playback volume 15 of the audio stream and / or the detected internal sound 37.

[0075] As mentioned above, speech enhancement due to background noise is not necessary if the internal noise 40' is below the masking threshold T for all frequencies. This can result, for example, in low ambient noise environments, or, for example, when hearing impairment compensation results in signal levels whose corresponding threshold T is above the internal ambient noise 40'.

[0076] In the following embodiment, the hearing impairment is determined by the user hearing profile HL(f k ) is calculated based on the speech intelligibility index known as the count-the-dots audio form (Note 4). A further optimization problem may be formulated using the hearing profile HL(f k ), as given by the count-the-dot audiogram, select a filter H to adjust the hearing threshold to maximize the standardized articulation index AI or speech intelligibility index SII. HI(k) is desirable. The AI ​​should provide the hearing profile HL(f k ) is plotted on the defined diagram, the hearing profile HL(f k ), which is indicated by the dashed line in FIG.

[0077] In one embodiment, this is the gain h as a function of frequency bins. HI The set of (k) can be described as an optimization formula that is optimized to amplify / attenuate and redistribute the energy of the audio stream so that the intelligibility index is maximized.

number

[0078] Therefore, the filter H HI (k)=h HI (k) is therefore generated from the required gain at frequency k such that the total energy change due to the resulting filter in the target audio stream 20' is equal to gamma. In one embodiment, γ=1 corresponds to the energy redistribution illustrated in FIG. 6, and in another example, γ=2 if a constant gain of 2 is obtained across all frequencies. Gamma may be set depending on the amplification possibilities in the system 1, typically set to 1 (unity) as illustrated above if no amplification is allowed, and set to 2 if doubling is possible. Again, the same reasoning applies here as for the energy distribution due to background noise: amplification across all frequencies is not possible.

[0079] Figure 6 shows an example count-the-dot audiogram, where the dotted line is the hearing test result in dB HL (hearing level). The solid line is the result after the compensation filter has been applied. As can be seen, the AI ​​has increased, i.e., there are more dots below the solid line than below the dotted line. Filter H HISince it is desired to make the overall energy change provided by (k) equal to 1, some dots are lost. The results shown in Fig. 6 are valid for a sound pressure of 60 dB, i.e., normal conversation or listening level in a reasonable environment. The approximate sound pressure level of the target audio stream 20' when played by the transducer 35 is available, for example, by measuring the sound pressure level using the internal microphone 36, by pre-calibration of the volume steps and corresponding sound pressure levels of the system 1, and it is possible to adjust, for example, the hearing profile accordingly.

[0080] The VDC, as described above, selects the filter H based on the dynamics of the audio stream 20. HI In one optional embodiment, the filter H is implemented using a pre-calibrated table that makes (k) a fixed value that does not change. HI (k) is updated only when the volume setting, i.e., playback volume 15, of audio device 30 is changed. In one embodiment, the pre-calibrated table is HI (k), for example, a(k)H HI The pre-calibrated table contains a scaling factor a(k) for each volume step and frequency bin applied to (k). The size of the pre-calibrated table depends on the number of volume steps and frequency bins used in the calculation.

[0081] In one embodiment, the VDC calculates two hearing compensations in the frequency domain, one for each ear. In an additional embodiment, it may be configured to combine left-ear and right-ear compensations to provide a single hearing compensation suitable for both ears. This is important when the reproduction uses several transducers and sound from each transducer can physically reach each ear of the listener.

[0082] In another embodiment of a VDC implementation that can be very well combined with the above embodiment, the feedback microphone signal is filtered by a filter H based on the level of the target audio stream 20' when played by the transducer 35. HI This approach allows for more frequent compensation calculations. To avoid sudden transients in the audio signal, the filter H HI The update rate of (k) may be kept low, around 0.25 to 1 Hz.

[0083] While the count-the-dots audiogram is provided assuming a signal or speech level of 60 dB SPL, the results and method 100 herein can be very well scaled with sound pressure level, with accurate results. In a non-limiting example, a 10 dB increase in volume, corresponding to a 10 dB SPL, corresponds to the dotted line in FIG. 6, which is offset by 10 dB toward 0 dB HL on the hearing level scale y-axis. This means that with an increase in volume, speech intelligibility increases and the corresponding hearing impairment compensation is adjusted accordingly. Lower volume levels have similar examples. As one skilled in the art would deduce, if the dotted line is above the region with dots that includes the offset due to the increased signal level, no enhancement is necessary.

[0084] Other methods of scaling results and adjusting for hearing impairment based on count-the-dot audiograms and the teachings herein will now be apparent to those skilled in the art.

[0085] As mentioned above, the inventors behind the present disclosure recognized that the internal noise 40' can be accurately modeled based on the ambient noise 40 through machine learning. The internal noise 40' is known from ANC technology to be the external noise 40 filtered by the primary (acoustic) path. Here, the primary path describes the effect on the external noise 40 as it propagates from outside the headphones 30 into the ear cavity. The primary path is a critical (unknown) noise transfer function that must be found with high accuracy in real time for the ANC to work correctly, i.e., for the ANC technology to form the correct anti-noise signal to cancel (and therefore attenuate) the internal noise 40'. The real-time and accuracy requirements for ANC technology typically dictate that the ANC technology be implemented on dedicated hardware.

[0086] In the present invention, the real-time and accuracy requirements for estimating the internal noise 40' are much lower compared to ANC techniques, as will be described below. Furthermore, no dedicated hardware is required. Many of the same aspects known to those skilled in the art remain, for example, the estimation of the internal noise 40' must filter out echo from the near-end audio stream as it is rendered by the transducer 35, and instead of complex real-time adaptive filtering and calibration (as in ANC), a neural network is used to model the primary path, including the separation between noise, echo, and the near-end talker.

[0087] The neural network of the preferred embodiment is an RNN. RNNs are typically based on long short-term memory (LSTM) or gated recurrent units (GRU). In general, the feature vectors and output vectors of an RNN can be selected in a number of ways, and two choices, namely, the training data quality and the size of the RNN, along with the training of the RNN, determine the performance of the RNN's ability to output desired output data given input, i.e., feature data.

[0088] The size of an RNN is determined by the training results and resource constraints posed by its implementation in a real-time audio system. In general, the size, i.e., the number of units and hidden layers of the RNN, is a design choice, similar to the choice of the number of points in the FFT calculation. A typical size of an RNN is, for example, 200-300 units with 3-4 hidden layers. The computational demand of an RNN can be reduced by, for example, selecting a lower-order RNN, increasing the error on the output, and / or skipping RNN units to perform structural pruning.

[0089] It should be noted that the absolute level of the signal provided by the external microphone 5 is important for the application. Therefore, an approach for data augmentation can be adopted in which each training example is pre-filtered with a randomized generated second-order filter, thereby training the RNN to be robust to frequency variations due to frequency response tolerances of the external microphone 5 and variations in placement of the external microphone 5. The levels of the individual signals in the training examples are preferably varied for robustness to level variations.

[0090] It should be apparent that the above description of an RNN is merely one non-limiting example provided as one working example of an RNN. Those skilled in the art, after digesting the teachings herein, can certainly devise different examples that modify the feature set, output set, and / or training, e.g., dataset, loss function, optimization process, etc.

[0091] Feature extraction for RNNs may be provided by speech recognition theory, and the following embodiments should be considered as non-exhaustive examples of feature extraction that can be combined with each other in any order or imaginable configuration.

[0092] In one embodiment, the features include the discrete cosine transform of the logarithm of the energy per critical band of the signal provided by microphone 5. In one embodiment, the features include the spectrum of the signal provided by microphone 5, represented in critical bands as described in this disclosure. In a further embodiment, the features further include the average energy over the entire subframe. In one embodiment, the features include the delta change in amplitude between the current subframe and the previous subframe cepstral log coefficients covering at least up to 600 Hz, preferably at least up to 1000 Hz, thereby containing the vocal basis of typical speech with high certainty. In one embodiment, the features include a binary signal indicating whether a non-noise source signal is active, such as a simple level detector indicating whether a user is speaking or an echo from a transducer is present.

[0093] The output of the RNN is typically the mean energy value E of the internal noise 40' for each auditory filter band. v (b), b = 1,…, N B In one embodiment, the output of the RNN also includes a binary signal indicating noise or no noise, a second binary signal indicating a low level or a high level, and a third binary signal indicating that the near-end signal is active. While additional signals such as the described binary signals may not be directly used in the algorithm to calculate the optimal speech filter 50, those skilled in the art will readily recognize that the related outputs as described may help to obtain better results when training the RNN.

[0094] In training an RNN, one important aspect is the RNN's generalization ability so that it operates correctly even under conditions not used during training. Thus, the rainfall example is preferably constructed by a combination of background noise, e.g., a full-size car at 100 km / h and a cafeteria (Note 5), various levels of echo signals and near-end talk, and filtering as described above. The echo signals and near-end talk signals are preferably independent, i.e., the same speech is not present at the same time.

[0095] In one embodiment, the training ground truth consists of the noise (only) power per auditory filter based on the spectrum of the noise signal (only), divided into auditory filters, and referenced (measured) within the headphones and the ear cavity. This therefore includes the primary path, i.e., the path from outside the headphones to the headphones and the inside of the ear cavity. This is important because the headphones, when worn (depending on the type of headphones: in-ear, over-ear, and on-ear), have at least high-frequency attenuation for noise due to the acoustic seal. If the headphones also have active noise cancellation (typically operable in the 150-900 Hz frequency range), the noise outside the headphones will be significantly different from the noise inside the ear cavity.

[0096] Systems that can facilitate the presentation of noise, near-end speech, and echo while simultaneously recording internal noise 40' (ground truth) are industry standard, and the process is fully automated. In one non-limiting example, a scenario is initiated in which background noise is rendered from a multi-speaker setup in a measurement chamber, and headphones under test are placed on a head and torso simulator that records the internal noise as signals reaching microphones located in each ear simulator. Simultaneously, background noise is recorded on headphones 30 by external microphones 5. After the scenario is complete, each signal is conditioned, time-aligned, and converted into either a feature set or a ground truth set.

[0097] In summary, the use of machine learning and recurrent neural networks in modeling the internal noise 40' will result in noise estimation in the auditory band model, and removing near-end talk and echo without the use of complex voice activity detection or echo cancellation will model the primary path from outside the headphones to inside the headphones and ear cavity.

[0098] Several detailed implementations of different aspects of the audio filter 50 are presented throughout this disclosure. Regardless of how the audio filter 50 is determined, H vo An audio filter 50, as described above as (k), is applied 140 to the audio stream 20, which is a digital source signal. There may be multiple audio filters 50, each providing a target audio stream 20' that is rendered on the transducer 35. There are several approaches to processing the audio stream 20, and those skilled in the art will recognize several other approaches beyond the two examples below after reading this disclosure. In one embodiment, the audio filter 50 may be applied by transforming the frequency function into a finite impulse response filter 140, for example, if the phase response is not very important, or may be a symmetric impulse response filter that results in a linear phase. In one embodiment, the audio filter 50 may be applied by multiplication in the frequency domain using an overlap-and-add method 140 to avoid circular convolution when multiplying the frequency function.

[0099] In one preferred embodiment, audio device 30 includes at least one audio filter 50 for each transducer 35 .

[0100] In one embodiment, the audio filter 50 providing the target audio stream 20' is energy normalized. This can lead to high peak amplitudes in the time-domain signal. In a further embodiment, the target audio stream 20' is attenuated to ensure that its signal amplitude is not too high for the final signal format. The signal amplitude may then be converted to the correct format without distortion, for example, using a standard limiter or dynamic range controller DRC. Note that no additional processing is required apart from controlling the signal amplitude. The limiter and DRC can be other components of a digital audio system and are preferably included for hearing safety.

[0101] 7, some further non-exhaustive steps of determining 130 the audio filter 50 are described. The following embodiments may be combined with any of the other embodiments presented herein and are fully compatible with the detailed examples and calculations disclosed herein.

[0102] In one embodiment of method 100, determining 130 the audio filter 50 includes subtracting 132 the estimated internal noise 40' from the detected internal sound 37. This provides a true audio stream 37', which is what the target audio stream 20' actually sounds like to the user's ear. Therefore, based on the difference between the target audio stream 20' and the true audio stream 37', it is possible to update 138 the audio filter 50 based on this difference. This effectively creates a control loop that can ensure that the target audio stream 37 is actually what the user hears. This is beneficial because it allows the audio filter to be updated based on, for example, how the audio device is worn by the user and how well the audio device fits the user's ears.

[0103] In one embodiment of method 100, determining 130 the audio filter 50 includes comparing 134 the estimated internal noise 40' with one or more masking thresholds T. This essentially compares 134 the estimated internal noise 40', the dense dashed line, against the masking thresholds T, the dashed line in FIG. 4. The audio filter is then updated based on the comparison 134 to provide the target audio stream 20'. In a further optional embodiment of method 100, the masking thresholds T are calculated by performing a critical band analysis 137 of the audio stream 20. The critical band analysis 137 includes auditory masking by frequency spreading.

[0104] In one embodiment, the method 100 is configured to compensate for the playback volume 15 of the audio device 30 as described with reference to FIG. 5 . This includes determining 136 a playback phon based on the playback volume 15 in the step 130 of determining the audio filter 50. The playback phon may be determined from the playback phon by a transfer function that describes the transfer function of the sound through the transducer 35. In addition, the step 138 of updating the audio filter 50 is further based on equal loudness curves associated with the determined phon. In a further embodiment, the playback phon is further based on the detected 160 internal sound 37.

[0105] In addition to solving the problems presented above, the present invention enhances speech intelligibility substantially regardless of how a user of an audio device 30 chooses to carry the audio device 30. Typically, the transducer 35 of an audio device is configured to operate with a particular load. This load is in the form of an air cavity between the user and the transducer 35. If the audio device 30 is, for example, a pair of closed headphones, the air cavity is created by the headphones 30 being carried tightly and securely around the user's outer ears. However, because not all ears are the same and not all users carry their audio devices 30 in the same way, the load of the transducer 35 varies between users, thereby causing the audio device 30 to sound different. The present invention also solves this problem by detecting 160 different internal sounds 37 depending on how the audio device 30 is worn.

[0106] (Note explanation) Note 1: The Bark scale is a psychoacoustic scale. Note 2: Boyd, S., & Vandenberghe, L. (2009). Convex Optimization. Cambridge University Press. ETSI. (2012). Speech and multimedia Transmission Quality (STQ); Speech quality performance in the presence of background noise; Part 1: Background noise simulation technique and background noise database 202 396-1. ETSI Note 3:S3.5-1997, A. (1997). Methods for calculation of the Speech Intelligibility Index. ANSI. Note 4: https: / / www.audiologyonline.com / articles / 20q-aided-speech-intelligibility-index-23707 Note 5: ETSI. (2012). Speech and multimedia Transmission Quality (STQ); Speech quality performance in the presence of background noise; Part 1: Background noise simulation technique and background noise database 202 396-1. ETSI.

Claims

1. A method (100) for enhancing speech intelligibility of an audio stream (20) containing speech audio performed in real time by an audio device (30), comprising: Detecting (110) ambient noise (40); estimating (120) an internal noise (40') based on the ambient noise (40), the estimated internal noise (40') being an estimate of the noise perceived at the eardrum of a user of the audio device (30); determining (130) an audio filter (50) based on the estimated internal noise (40') and the audio stream (20), wherein the audio filter (50) is a psychoacoustic masking-based audio filter (50); applying (140) said audio filter (50) to said audio stream (20) to provide a target audio stream (20'); outputting (150) the target audio stream (20') to one or more transducers (35) to thereby generate an internal sound (37) at the eardrum of the user of the audio device (30); detecting (160) the internal sound (37) of the audio device (30); Including, determining (130) the audio filter (50) further based on the detected internal sound (37); subtracting (132) the estimated internal noise (40') from the detected internal sound (37) to provide a true audio stream (37'); updating (138) the audio filter (50) based on the difference between the target audio stream (20') and the true audio stream (37'); A method (100) comprising:

2. The step (130) of determining the audio filter (50) comprises: a step (134) of comparing the estimated internal noise (40') with one or more masking thresholds (T); updating (138) the speech filter (50) based on the comparison (134); further comprising 2. The method (100) of claim 1, wherein the one or more masking thresholds (T) are calculated by performing a critical band analysis (137) of the audio stream (20), the critical band analysis (137) including auditory masking by frequency spreading.

3. A hearing profile (HL(f)) associated with a user of the audio device (30) ｋ ) filtering (105) the audio stream (20) to compensate for The method (100) of claim 1, wherein the step (130) of determining the audio filter (5) is preferably performed after the filtering (105), such that the determination (130) is based on an audio stream (20) compensated for a hearing profile (HL(f k )) associated with the user of the audio device (30).

4. The step (130) of determining the audio filter (50) comprises: determining (136) the playback phone based on the playback volume (15); further comprising The step (138) of updating the audio filter (50) is further based on the determined equal loudness contours associated with the playback phone, 2. The method (100) of claim 1, wherein the step (136) of determining the playback phone is further based on the internal sound (37).

5. 2. The method of claim 1, wherein the step of determining the audio filter further comprises smoothing the gain of the audio filter in frequency by convolution with a frequency window function.

6. The step of determining (130) the audio filter (50) includes one or more weighting parameters (T ｉ 2. The method of claim 1, further comprising averaging the gain of the speech filter using an exponentially weighted moving average including:

7. 2. The method of claim 1, wherein the step of determining the audio filter further comprises applying a configurable blending setting to select the degree to which the audio filter should be applied to the audio stream.

8. The method (100) according to any one of claims 1 to 7, wherein the step (120) of estimating the internal noise (37') is implemented by one or more recurrent neural networks (RNN).

9. The ambient noise (40) is detected by an external microphone (5) operatively connected to the audio device (30); and / or The method (100) according to any one of claims 1 to 7, wherein the ambient noise (40) is limited to a maximum audio bandwidth of up to 10 kHz, preferably up to 8 kHz.

10. The method (100) of any one of claims 1 to 7, further comprising applying active noise cancellation (ANC) to the audio stream (20) after applying (140) the audio filter (50) to the audio stream (20).

11. An audio device (30) comprising one or more transducers (35), at least one internal microphone (36) arranged to detect internal sounds (37) in a user's ear cavity, and a processing module (32) operatively connected to the internal microphone (36), the one or more transducers (35), and an external microphone (5), wherein the processing module (32) is configured to perform the method (100) of any one of claims 1 to 7.

12. 12. The audio device (30) of claim 11, wherein the external microphone (5) is included in the audio device (30).

13. 12. An audio system (1) for enhancing speech intelligibility in real time, the system (1) comprising: a portable electronic device (10) operably connected to an audio device (30) and configured to transmit an audio stream (20) containing speech audio to the audio device (30), the audio device being the audio device of claim 11.

14. 14. The audio system (1) of claim 13, wherein the ambient noise (40) is sensed by an external microphone (5) included within the electronic device (10), and the electronic device (10) is further configured to transmit the ambient noise (40) sensed by the external microphone (5) to the audio device (30).

15. When executed by a processing module (32), said processing module (32) A computer program product (200) adapted to cause a computer to perform the method (100) of any one of claims 1 to 7.

Images