Method for improving speech intelligibility through context adaptation

Abstract

The method includes the steps of detecting noise in an environment using a microphone to generate a noise signal, receiving an audio signal to be played in the environment via a loudspeaker, performing multiband correction of the noise signal based on a microphone transfer function of the microphone to generate a corrected noise signal, performing multiband correction of the audio signal based on a loudspeaker transfer function of the loudspeaker to generate a corrected audio signal, and calculating a multiband speech intelligibility result based on the corrected noise signal and the corrected audio signal.

Description

[Technical field]

[0001] (Priority Claim) This application claims priority to U.S. Provisional Application No. 62 / 898,977, filed September 11, 2019, which is incorporated herein by reference in its entirety. [Background technology]

[0002] (Technical field) The present disclosure relates to speech intelligibility processing. Summary of the Invention [Problem to be solved by the invention]

[0003] Audio reproduction devices, such as artificial intelligence (AI) speakers, mobile phones, teleconferencing, Internet of Things (IoT) devices, and others, are often used in acoustic environments with high levels of background noise. The audio reproduced by the audio reproduction device may be masked by the background noise, resulting in reduced audio intelligibility. Many techniques are available to improve audio intelligibility. Some of the techniques also use noise capture devices to improve audio intelligibility in noise. However, these techniques do not clarify and address practical challenges associated with specific implementation limitations, such as the physical limitations of the reproduction device, the physical limitations of the noise capture device, signal headroom for audio intelligibility processing, and audio characteristics over time. [Means for solving the problem]

[0004] By addressing the above-mentioned issues and problems, optimal performance of natural transition from unprocessed to processed speech can be achieved. Thus, the embodiments presented herein introduce novel features and improvements for speech intelligibility analysis that improve speech intelligibility in noisy environments and overcome the practical issues described herein. The embodiments include, but are not limited to, (1) digital-to-sound level conversion combined with multi-band speech and noise compensation, (2) short-segment speech intelligibility analysis, (3) long-segment speech and noise profiling, and (4) global and per-band gain analysis. Since the analysis results performed in the embodiments provide relative gain adjustment parameters for wideband and per-frequency bands of the audio signal for playback, the processing in the embodiments is not limited to a specific audio signal processing, but may include any combination of known dynamic processing such as compressor, expander, formant enhancement, etc. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a high-level block diagram of an exemplary system in which embodiments directed to speech intelligibility processing may be implemented. [Diagram 2] FIG. 2 is a block diagram of an exemplary voice intelligibility processor (VIP) and associated speech and noise processing implemented in the system of FIG. 1. [Diagram 3] 1 is an exemplary plot of a band importance function of the Speech Intelligibility Index (SII). [Figure 4] FIG. 2 illustrates example loudspeaker frequency responses for two different loudspeakers. [Diagram 5] 1 is a frequency plot for various frequency analysis ranges determined based on the frequency response of an idealized exemplary microphone and the frequency response of an idealized exemplary loudspeaker, and the interrelationship between the two frequency responses. [Figure 6] 1 is a plot of a short segment of an audio signal and its corresponding frequency spectrum. [Figure 7] 4 is a plot of another short segment of an audio signal and its corresponding frequency spectrum. [Figure 8] 1 is a plot of a long segment of an audio signal and its corresponding frequency spectrum. [Figure 9] FIG. 2 is a high level block / signal flow diagram of a portion of the VIP Voice Enhancer. [Figure 10] 1 is a flow chart of an example method of multi-band speech intelligibility analysis / processing and speech intelligibility enhancement performed by a VIP. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As used in this specification, the terms "speech", "voice", and "voice / speech" are synonyms and can be used interchangeably, "frame", "segment", and "time segment" are synonyms and can be used interchangeably, "speech (or voice) intelligibility" and "intelligibility" are synonyms and can be used interchangeably, "bin" and "band" are synonyms and can be used interchangeably, and "bandwidth (BW)" and "pass band" are synonyms and can be used interchangeably.

[0007] FIG. 1 is an exemplary system 100 capable of implementing the embodiments presented herein. The system 100 is an example, and many variations are possible. Such variations may omit or add audio components. The system 100 may represent an audio communication device supporting audio communication, e.g., audio calls, with a remote communication device not shown. The system 100 may also represent a multimedia playback device coupled with a communication device. Non-limiting examples of the system 100 include a telephone (e.g., a mobile phone, a smart phone, a Voice over Internet Protocol (IP) (VoIP) phone, and the like), a computer (e.g., a desktop computer, a laptop, a tablet, and the like), and a home theater sound system with an audio communication device.

[0008] The system 100 is deployed in an acoustic environment, such as a room, an open space, or a similar environment. The system 100 includes an audio transmission path, an audio playback path, and a media playback path coupled together. The audio transmission includes a microphone 104, an acoustic echo canceller 106, and a noise preprocessor 108, which represent an audio / noise capture device (also referred to simply as a "noise capture device") coupled together. The microphone 104 converts a sound in the acoustic environment into a sound signal representative of the sound. The sound signal represents background noise of the acoustic environment (also referred to simply as "noise") and may also represent a voice from a speaker. The acoustic echo canceller 106 and the noise preprocessor 108 (collectively hereinafter "preprocessors") respectively cancel echoes and reduce noise in the sound signal and transmit the processed sound signal (e.g., processed voice) for playback, for example, at a remote station.

[0009] The audio playback path includes a voice intelligibility processor (VIP) 120, a system volume control 122, and a loudspeaker 124 (more generally, a playback device). In the audio playback path, the VIP 120 receives an audio signal (i.e., an audio playback signal) to be played via the speaker 124. For example, the audio signal can be transmitted to the system 100 from a remote communication device (e.g., a remote mobile phone) as mentioned above for playback. In addition, the VIP 120 receives a noise signal from the microphone 104, which is representative of the noise of the acoustic environment. The noise signal received by the VIP 120 can be an echo-canceled noise signal generated by the acoustic echo canceller 106 to avoid self-activation of the VIP. According to the embodiment presented herein, the VIP 120 simultaneously processes the audio signal for playback together with the noise signal (e.g., the noise sensed by the microphone 104) to generate an intelligibility-enhanced audio signal in order to enhance the intelligibility of the audio signal. The VIP 120 provides an intelligibility-enhanced audio signal (via a system volume control 122) to a loudspeaker 124 for playback by the speakers into the acoustic environment.

[0010] The media playback path includes an audio post-processor 130, a system volume control 122, and a loudspeaker 124. The audio post-processor 130 processes the media signal (via the system volume control 122) for playback by the speaker 124. The system 100 may also include a switch 140 that selectively directs either audio playback or media playback to the loudspeaker 124.

[0011] The system 100 also includes a controller 150 coupled to the microphone 104 and the loudspeaker 124. The controller 150 can be configured to implement, for example, the acoustic echo canceller 106, the noise pre-processor 108, the VIP 120, the audio post-processor 130, the switch 140, and the system volume control 122. The controller 150 includes a processor 150a and a memory 150b. The processor 150a can include, for example, a microcontroller or microprocessor configured to execute software instructions stored in the memory 150b. The memory 150b can include a read-only memory (ROM), a random access memory (RAM), or other physical / tangible (e.g., non-transitory) memory storage device. Thus, in general, the memory 150b can include one or more computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, which, when executed (by the processor 150a), are operable to perform the operations described herein. For example, memory 150b stores or encodes control logic instructions for implementing VIP 120 (e.g., the VIP modules described below in connection with Figures 2-9) and other modules of system 100 described above and for performing overall control of system 100.

[0012] The memory 150b also stores information / data 150c used and generated by the control logic as described herein.

[0013] 2 is an exemplary high-level block diagram of the VIP 120 and the processing performed by the VIP, according to one embodiment. The VIP includes a voice and noise analyzer 202 coupled to a voice enhancer 204. The voice and noise analyzer 202 receives a noise signal from the microphone 104. The voice and noise analyzer 202 also receives an audio signal for playback. In one example, the noise signal and the audio signal are time-domain signals, each of which may be in a pulse code modulation (PCM) format, although other formats are possible. The voice and noise analyzer 202 simultaneously analyzes and processes the noise signal and the audio signal to generate a multi-band speech intelligibility result 205, which is provided to the voice enhancer 204. The voice enhancer 204 processes the audio signal based on the multi-band speech intelligibility result 205 to enhance or improve the intelligibility of the audio signal and generate an intelligibility-enhanced audio signal. The intelligibility-enhanced audio signal is played back via the system volume control 122 and the loudspeaker 124.

[0014] The speech and noise analyzer 202 includes a noise compensation path 206, a speech compensation path 208, a speech intelligibility calculator 210 following the two compensation paths, and a gain determiner 212 following the speech intelligibility calculator 210. The noise compensation path 206 includes a noise digital acoustic converter (DALC) 222 and a multiband noise compensator 224 following the noise DALC. The speech compensation path 208 includes a speech DALC 226 and a multiband speech compensator 228 following the speech DALC. The speech intelligibility calculator 210 includes a short segment analyzer 230, a long segment analyzer 232, and a silence / pause detector 234. The noise compensation path 206 receives pre-measured and / or derived noise pickup device parameters 240 (e.g., known microphone parameters) that characterize or relate to the microphone 104. The audio correction path 208 receives pre-measured and / or derived reproduction device parameters 242 (eg, known loudspeaker parameters) that characterize or relate to the loudspeaker 124 .

[0015] At a high level, the noise compensation path 206 applies multi-band noise compensation to the noise signal based on the noise pickup device parameters 240. Specifically, based on the noise pickup device parameters 240, the noise DALC 222 performs digital-to-sound level conversion (e.g., scaling) of the noise signal, and the noise compensator 224 performs multi-band noise compensation on the converted or scaled noise signal to generate a corrected noise signal. The noise compensation path 206 provides the corrected noise signal to the speech intelligibility calculator 210. Similarly, the speech compensation path 208 applies multi-band speech compensation to the speech signal. Specifically, based on the playback device parameters 242, the speech DALC 226 performs digital-to-sound level conversion (e.g., scaling) of the speech signal, and the speech compensator 228 performs multi-band compensation on the converted / scaled speech signal to generate a corrected speech signal. The speech compensation path 208 provides the corrected speech signal to the speech intelligibility calculator 210.

[0016] The speech intelligibility calculator 210 performs a multi-band speech intelligibility analysis on the corrected noise signal and the corrected speech signal to generate and provide a multi-band speech intelligibility result (MVIR) to the gain determiner 212. More specifically, the short segment analyzer 230 performs a multi-band speech intelligibility analysis on the short / medium length frames / segments of the corrected noise / speech to generate a multi-band speech intelligibility result for the short / medium length segments (also referred to as "short-term speech intelligibility result" or simply "short-term result"). The short-term result includes a sequence of speech intelligibility values ​​per band, global speech intelligibility values, noise power values ​​per band, and speech power values ​​per band corresponding to the sequence of short / medium length segments of noise / speech.

[0017] Meanwhile, the long segment analyzer 232 performs long-term noise and speech profiling (including speech intelligibility analysis) on long frames / segments of the corrected noise / voice that are longer than the short / medium length segments to generate long segment speech intelligibility results (also referred to as "long-term speech intelligibility results" or simply "long-term results"), such as long-term per-band speech intelligibility values ​​and long-term global gain values. For example, the long-term noise and speech profiling may perform a running average of values ​​(e.g., over a time period as long as the long segment) in a sequence of short-term results to generate the long-term results. The long-term noise and speech profiling may also employ other types of long-term processing of the short-term results, such as, for example, peak-holding and resetting of noise / speech power values ​​over multiple short / medium length segments, as described below.

[0018] A silence / pause detector 234 detects silence / pauses in the corrected speech signal to suspend intelligibility analysis during silence, eg, to prevent intelligibility analysis from being initiated during silence.

[0019] The speech intelligibility results provided to the gain determiner 212 may include a combination of short-term and long-term results. The gain determiner 212 derives global and per-band gains for short / medium length segments based on the aforementioned speech intelligibility results and provides the gains to the voice enhancer 204. The voice enhancer 204 may include a voice compressor, a voice expander, a formant enhancer, etc. The voice enhancer 204 performs a voice enhancement process on the (uncorrected) speech signal based in part on the analysis results 205. For example, the voice enhancer 204 applies a gain to the speech signal to generate an intelligibility-enhanced speech signal, which is played via the system volume control 122 and the loudspeaker 124.

[0020] The embodiments illustrated herein include, but are not limited to, multi-band noise and speech correction performed by noise and speech correction paths 206, 208, short / medium length segment speech intelligibility analysis performed by short segment analyzer 230, long term noise and speech profiling performed by long segment analyzer 232, and global and per band gain analysis performed by gain determiner 212. The embodiments are described in more detail below.

[0021] (Multi-band noise and audio correction) Multiband noise and speech analysis is known. One form of such analysis includes the Speech Intelligibility Index (SII). The SII analysis receives a multiband speech signal reproduced in an acoustic environment via a loudspeaker and a noise signal representative of the noise of the acoustic environment detected by a microphone. The SII analysis (i) calculates the difference in the levels of the speech signal and the noise signal per frequency band of the speech signal, e.g., calculates a speech-to-noise ratio (SNR) per frequency band of the speech signal, multiplies the SNR per band by a band importance function for the corresponding frequency band, and sums the results.

[0022] Figure 3 shows different plots of band importance function of speech intelligibility index. Band importance function basically gives different weights to frequency bands of the audio signal according to their contribution / importance to the speech / voice intelligibility. Along with band importance function, it has been studied that fundamental and first formant of human speech may have less impact on speech / voice intelligibility compared to second formant and other factors. These are important factors to consider when calculating speech intelligibility.

[0023] Direct manipulation of the frequency response of the audio signal based on the intelligibility index and the intelligibility contribution factors of each frequency band may degrade the audio quality when played through a loudspeaker. For example, such manipulation may cause the audio to sound unnatural when changing the frequency balance and / or introducing time-varying variations. In addition, if the frequency response of the transducers (e.g., the frequency response of the microphone and the loudspeaker) is not corrected before the intelligibility analysis described above, the results of the intelligibility analysis (e.g., SII) will not be accurate. Furthermore, if the loudspeaker is unable to reproduce the full frequency bandwidth of the audio due to the constraints of the loudspeaker (e.g., its small size or small driver), the loudspeaker may further degrade the audio / speech quality at frequency balance changes, resulting in inaccurate audio intelligibility results. Increasing the gain of the audio frequencies that the loudspeaker cannot reproduce does not solve the problem and may cause nonlinear distortion and / or strain the loudspeaker driver.

[0024] The frequency response of two different loudspeakers (spk1 and spk2) is shown in Figure 4. Since transducer characteristics differ from loudspeaker to loudspeaker and from microphone to microphone, the loudspeaker and microphone compensation of a given system needs to be taken into account when calculating multiband speech intelligibility.

[0025] Thus, the multiband noise and speech compensation performed by the noise and speech compensation paths 206, 208 compensates for the noise and speech frequency bands used in computing the multiband speech intelligibility result based on the respective characteristics of the loudspeaker and microphone in addition to the intelligibility contribution factors of each of the frequency bands. As an example, the noise compensation path 206 compensates for the noise signal (H ns ) frequency bands (e.g., by adjusting the power level of the frequency bands) to obtain a compensated noise signal (H An_ns ), and the audio correction path 208 generates an audio signal (H spch) frequency bands (e.g., by adjusting the power level of the frequency bands) to produce a compensated speech signal (H An_spch ). The speech intelligibility calculator 210 then generates the corrected noise signal (H An_ns ) and the corrected audio signal (H An_spch ) is subjected to multi-band speech intelligibility analysis.

[0026] An example of a noise pickup device parameter 240 is the microphone transfer function H mic (e.g., a known microphone transfer function), the gain associated with the microphone, g mic (i.e., the output gain of the noise signal), the acoustic-to-digital conversion gain of the noise signal, c mic , and microphone sensitivity. Examples of playback device parameters 242 include the loudspeaker transfer function H spk (i.e., the known loudspeaker transfer function), the gain associated with the loudspeaker, g spk (i.e., the output gain of the audio signal), and the audio-to-digital conversion gain c spk , and loudspeaker sensitivity (which may be provided separately or incorporated into other parameters). The transfer function may include a frequency domain representation of the time domain impulse response of the corresponding transducer (e.g., microphone or loudspeaker), including amplitude and phase information across multiple contiguous frequency bands spanning the transfer function. As an example, the audio correction path 208 uses the playback device parameters 242 to adjust the audio signal (H spch ) (e.g., the frequency spectrum of an audio signal) and calculate the z-domain corrected audio signal (H An_spch ) H An_spch (z)=H spch (z)*H spk (z)*g spk *c spk Formula (1)

[0027] For example, the audio DALC226 has a parameter g spk and c spkand the audio compensator 228 scales the audio signal based on the loudspeaker transfer function H spk (z) performs multiband correction of the scaled audio signal based on (z).

[0028] Similarly, the noise compensation path 206 uses the noise pickup device parameters 240 to calculate the noise signal (H ns ) and correct the z-domain corrected noise signal (H An_ns ) H An_ns (z)=H ns (z)*H mic (z) -1 *g mic *c mic Formula (2)

[0029] For example, the noise DALC222 has a parameter g mic and c mic and the noise compensator 224 scales the noise signal based on the microphone transfer function H mic Based on (z), we perform multiband correction of the scaled noise signal, which gives us an accurate estimate of the noise in the acoustic environment.

[0030] The above-mentioned scaling of the noise and speech signals may include scaling based in part on microphone sensitivity and speaker sensitivity, respectively. In one example, the scaled noise / speech values ​​are given by: Scale_val=10 (A / 20) / 10 (D / 20) =10 ((A-D) / 20) where A = acoustic level (dB) and D = equivalent digital level (dB).

[0031] Such scaling is performed separately for the microphone 104 and loudspeaker 124 to match the respective input signals (i.e., noise or speech) to corresponding sound levels (dB). Alternatively, scaling can be done to match noise and speech levels relative to the microphone and loudspeaker sensitivities. Subsequent speech intelligibility calculations performed on the scaled values ​​use the ratio of the (corrected) speech signal to the (corrected) noise signal from the same acoustic environment, so that the intelligibility calculations are accurate when the deltas caused by different microphone and loudspeaker sensitivities are adjusted for.

[0032] in this case: Scale_val_mic=10 (Aspk / 20) / 10 (Amic / 20) Here, A spk and A mic is the measured / calculated sound level (dB) based on the same level digital level (dBFS). Scaling adjusts for the relative delta so that the scaled value can only be applied to noise signals. Alternatively, the inverse of Scale_val_mic can only be applied to speech signals.

[0033] The compensation of the speech signal and the noise signal in Equation (1) and Equation (2) improves the subsequent multiband speech intelligibility analysis. In addition to the speech and noise compensation, the embodiment shown herein performs a multiband (frequency) domain analysis on the frequency response of the microphone 104 and the loudspeaker 124. The multiband domain analysis can be performed in the noise compensation path 206, the speech compensation path 208, and / or the speech intelligibility calculator 210, or by a separate module of the speech and noise analyzer 202. The multiband analysis investigates / determines the overlapping and non-overlapping interrelationships between the frequency bands of the microphone and the frequency bands of the loudspeaker, and divides the frequency bands used for the multiband speech intelligibility analysis into different frequency analysis domains / ranges based on the determined interrelationships. The multiband speech intelligibility analysis is then performed based on (i.e., taking into account) the different frequency analysis domains set by the multiband domain analysis. For example, the multiband speech intelligibility analysis can apply different types of intelligibility analysis to the speech analysis bands in different frequency analysis ranges, as described below.

[0034] 5 shows an idealized (brick-wall) microphone frequency response 502 and an idealized loudspeaker frequency response 504, along with frequency plots of various frequency analysis ranges (a)-(g) determined by multiband domain analysis based on the interrelationship between the two frequency responses. The microphone frequency response 502 is expressed as the minimum ("min") / start frequency f mic1 A useful / responsive microphone frequency range or bandwidth (BW) / frequency passband (e.g., 3 dB BW, although other measures of what is considered a useful microphone passband may be used) extending from a minimum / start frequency f of the loudspeaker frequency response. spk1 to maximum / stop frequency f spk2

[0033] The loudspeaker has a useful / responsive loudspeaker frequency range or BW / frequency passband (e.g., 3 dB BW, although other measures may be used to consider the useful loudspeaker passband) that extends to

[0035] In the example of Figure 5, the minimum or starting frequency f spk1 ,f mic1 f spk1 >f mic1 The maximum or stop frequency fmic2,f spk2 fmic2>f spk2 Thus, the microphone passband exceeds and completely encompasses the loudspeaker passband, i.e., the loudspeaker passband is entirely within the microphone passband. In this case, the loudspeaker and microphone passbands overlap only over the loudspeaker passband. In another example, the reverse can also be true, i.e., the minimum frequency is f mic1 >f spk1 , with a maximum frequency of f spk2 >fmic2, such that the loudspeaker passband exceeds and completely encompasses the microphone passband, i.e., the microphone passband is completely within the loudspeaker passband. In this case, the loudspeaker and microphone passbands overlap only at the microphone passband.

[0036] In the example of FIG. 5, the multiband region analysis can classify frequency analysis regions (a) to (g) (simply referred to as “regions (a) to (g)”) as follows with respect to performing multiband speech intelligibility in these regions: Regions (a) and (b) can be defined as regions that are unchanged by speech intelligibility analysis, or as regions of attenuation for headroom conservation, i.e. to preserve headroom. b. Areas (c) and (g) should not be included in the speech intelligibility analysis because the noise capture device (e.g., microphone) cannot provide accurate analysis results. mic1The frequency range below fmic2 and above fmic2 is H mic The reciprocal of (H mic -1 ) contains unstable capture frequency regions / bands that are not stable enough to be applied to the noise signal for noise correction. c. Regions (d) and (f) should be included in the speech intelligibility analysis for calculating the (global) noise level and masking threshold, but are not included in the per-band speech intelligibility analysis, e.g. any per-band speech level increases in regions (d) and (f) resulting from the speech intelligibility analysis cannot be accommodated by a playback device that is not responsive in these regions. d. If the arrangement of the loudspeaker and microphone frequency response is reversed from that of FIG. 5, i.e., the loudspeaker passband exceeds the microphone passband, then the frequency response in region (d) (i.e., f spk1 and f mic1 The noise signal level in the frequency band adjacent to the region (e.g., between f mic1 The noise signal level can be approximated using the noise signal levels above / below (above / below). In this case, the corrected noise signal can be calculated as follows: H An_ns (k)=alpha*H An_ns (k+1) Here, α is an approximation coefficient ranging from 0 to 1.0, and the minimum value is preferably greater than 0.

[0037] In the example of Figure 5, where the microphone passband is wider than and encompasses the loudspeaker passband, the level of the noise signal after applying the correction of equation (2) to the noise signal is considered accurate, so regions (d) and (f) should be included in the global noise level and masking threshold calculation. However, in the alternative / reverse example, where the loudspeaker passband is wider than the microphone passband and encompasses the microphone passband, regions (d) and (f) are treated differently, since the level of the speech signal is accurate, but the level of the noise signal within the regions is not. In this case, regions (d) and (f) can be excluded from both the global and per-band analysis.

[0038] Taking into account the frequency analysis range as described above improves the accuracy of the speech intelligibility analysis since frequency bands with inaccurate noise levels are excluded from the analysis, and the speech intelligibility analysis provides optimal global and per-band speech intelligibility results by dealing with differences in loudspeaker and microphone frequency ranges / passbands.

[0039] The speech and noise compensation can then be combined into an intelligibility contribution factor per frequency band (i.e., per speech analysis band). For example, the speech / noise compensation can be used to produce a per-band (speech) intelligibility value V idx (i) (for frequency band i=1 to N) can be calculated as follows: V idx (i)=I(i)*A(i), i=from max(f mic1 ,f spk1 ) to min(f mic2 ,f spk2 ) Eq.(3) Where: i = band index identifying a given band (e.g., band i = 1 to band i = 21). I=Importance A = band audibility, and The function max(f mic1 ,f spk1)~min(f mic2 ,f spk2 ) determines / defines the frequency overlap between the loudspeaker and microphone passbands (eg, "overlapping passbands" where the loudspeaker and microphone passbands overlap).

[0040] The speech and noise analyzer 202 uses the above relationship to determine the overlapping passbands based on the start and stop frequencies of the loudspeakers and microphones.

[0041] The band audibility value A is based on the corrected speech signal and the corrected noise signal from equations (1) and (2), respectively. For example, the band audibility value A can be proportional to the ratio of the power of the corrected speech signal to the power of the corrected noise signal in a given band. The frequency analysis range per band is defined / corrected based on the noise pickup device parameters 240 and the playback device parameters 242 described above.

[0042] From the above, equation (3) generates speech intelligibility results from speech analysis bands 1 to N based on different frequency analysis regions as follows: a. Band 1 (i.e., the lowest frequency band) to max(f mic1 ,f spk1 )⇒Understanding level N / A. bf spk1 From f spk2 Up to => The speech intelligibility per band is given by equations (1) and (2). c.min(f mic2 ,f spk2 ) to Band N (i.e., the highest frequency band) ⇒ Intelligibility N / A.

[0043] max(f mic1 ,f spk1 ) is f spk1 In this case, the region (a) shown in FIG. 5 can be attenuated to provide processing headroom. max(f mic1 ,f spk1 ) is fmic1 If f spk1 The area below the maximum output level can be used to provide headroom. This headroom can be important when the audio signal reaches (or is close to) the maximum output level of such a system (e.g., a loudspeaker). In this case, there is no headroom for speech intelligibility analysis, so intelligibility cannot be improved. Alternatively, a compressor / limiter can be introduced to increase the root mean square (RMS) value of the audio signal while maintaining its peak value, but compression artifacts such as unnatural sounds and "pumping" can occur if the amount of compression exceeds a certain level. Thus, if a loudspeaker cannot fully reproduce a certain frequency range in an area, the audio signal in that area can be attenuated to provide headroom.

[0044] Using the speech correction and its analysis domain calculations, a global speech intelligibility value (also called global speech-to-noise ratio (SNR) (Sg), or equivalently called global speech-to-noise ratio) can be calculated according to the following formula:

number

[0045] According to the above relationship, frequency C1 = max(fmic1 ,f spk1 ), frequency C2=min(f mic2 ,f spk2 ), and frequency range C1-C2 is the overlap frequency range (i.e., overlap passband) between the microphone passband and the loudspeaker passband. The numerator of equation (4) accumulates / sums the corrected sound power only in the overlap frequency range, and the denominator accumulates / sums the corrected noise power only in the microphone frequency range / passband.

[0046] The short segment analyzer 230 generates a sequence of per-band speech intelligibility values ​​calculated according to equation (3) and a sequence of global speech-to-noise ratios (Sg) calculated according to equation (4). The long segment analyzer 232 processes (e.g., averages) the stored values ​​(i.e., sequences of values) of noise power and speech power from the short segment analyzer 230 over a number of short / medium length segments equal to a long segment to generate per-band intelligibility values ​​for the long segment and a global intelligibility value for the long segment. The long segment analyzer 232 can perform further operations on the short-term stored values, such as peak hold and reset, as described below.

[0047] The embodiment described above in connection with FIG. 5 determines the analysis region that is used as the frequency range setting or limit for equations (3) and (4). In another embodiment, limits are included in each weighting factor so that each weighting factor essentially calculates equations (3) and (4) without limiting the range. An_spch and H An_ns In this embodiment, the correction is applied as follows: H An_spch =W sp *H An_spch H An_ns =W ns *H An_ns Here, W sp and Wns are the weighting coefficients for speech and noise applied to each frequency band (0 to π).

[0048] In summary, the embodiments provide a comprehensive method for calculating noise / speech intelligibility with noise / speech compensation as follows. a. The characteristics of the playback and noise capture devices are used to calibrate the speech and noise signals and to define speech and noise frequency bands or ranges for analysis. b. Consider the weighting factor of the speech intelligibility contribution versus the frequency range of the device. c. Given a speech and noise input to the speech and noise analyzer 202, an analysis is performed to generate a speech intelligibility value having processing gain parameters for each band and / or a global processing gain value.

[0049] It should be noted that the analysis described herein is not limited to a fixed frequency band, but rather the frequency band may be an octave band, a 1 / 3 octave band, a critical band, and the like.

[0050] (Short / Medium Length Segment Speech Intelligibility Analysis) Many speech playback use cases require minimal latency. Therefore, it is not practical to use long segments of about 1 second or more (e.g., speech / noise long segments) for speech intelligibility analysis (referred to as "long segment analysis"), since long segment analysis may result in excessive latency. In contrast, short / medium length segments for speech / noise analysis and processing are typically of time duration of about 2-32 ms. Also, noise may not be static but dynamic, e.g., dog barking, loud car passing, etc. Therefore, multiband speech intelligibility analysis of short / medium length segments (referred to as short / medium length segment analysis) that are relatively shorter than long segments is preferred. That is, analysis of short / medium length segments is preferred over long segments.

[0051] The problem with short / medium length segment analysis is that when combined with other processing (e.g. gain processing), short / medium length segment analysis can produce unwanted artifacts. For example, if the processing gain adapts too quickly, the frequency balance of the audio may change too frequently, as well as introducing unnatural audio fluctuations. A common way to reduce such artifacts is to add smoothness to the gain changes by setting attack and decay times.

[0052] However, smoothing the speech intelligibility results in this way trades off accuracy and stability. For optimal accuracy while maintaining a stable sound, longer term speech noise profiling can be used to improve performance. The embodiments presented herein differ from conventional methods by combining traditional short / medium length segment analysis with long term speech noise profiling as shown below.

[0053] (Long-term speech and noise profiling) In contrast to short / medium length segments of 2-32 ms, the long segments analyzed by long-term speech and noise profiling can be from 2 words to a few sentences in length (e.g., about 1-30 seconds). Long-term speech and noise profiling does not require long-term storage of the noise / speech signal. Rather, it accumulates short-term results (i.e., characteristics of short / medium length segments) over time (i.e., over the long segments) in a sliding window. Because the long-term analysis uses past samples of speech and noise, the long-term analysis obtained from long-term speech and noise profiling does not increase the latency of the speech intelligibility results.

[0054] Figures 6, 7 and 8 show different time segments of a speech signal and their corresponding frequency spectra. Figure 6 includes an upper plot showing a short time segment (i.e., a "short segment") of the speech signal and a lower plot showing the frequency spectrum of the short segment. The short segment includes 1024 speech samples over a short segment of approximately 23 ms. Similarly, Figure 7 includes an upper plot showing another short segment of the speech signal and a bottom plot showing a second frequency spectrum of the short segment. The short segments shown in the upper plots of Figures 6 and 7 are each periodic, as is typical of speech. The frequency spectra shown in the bottom plots of Figures 6 and 7 are different because different phonemes have different formant frequencies.

[0055] FIG. 8 includes a top plot showing a long time segment (i.e., "long segment") of a speech signal and a bottom plot showing the frequency spectrum of the long segment. The long segment includes 1024 speech samples spanning approximately 4.24 seconds. The short segments of FIGS. 6 and 7 and the long segment of FIG. 8 capture common data including the speech fundamental frequency, but the long segment shows the spectral characteristics of speech over a longer period of time. Thus, speech intelligibility analysis including long-term speech and noise profiling can benefit from a wider frequency band analysis value and capture the long-term characteristics of the speech signal over the long segment, rather than just attempting to dynamically allocate narrow band frequency gains based on per-band analysis, which may change rapidly over time. In addition, long-term speech and noise profiling also captures the temporal characteristics of speech over the long segment.

[0056] Examples of consistent noise in an environment include fan noise or hum combined with occasional transient / dynamic noise such as dogs barking and cars passing. In this case, long-term speech and noise profiling can identify the static / consistent noise features, while short / medium length segment analysis can identify the dynamic noise. Long-term speech and noise profiling can capture peak noises that can be reset by comparing long-term and short-term results to identify if the consistent background noise has changed or been removed. For example, long-term speech and noise profiling can include peak-holding long segments of speech / noise, but then using short-term results to determine whether to reset the peaks when, for example, the speech playback changes to another speaker or a synthetic voice. Another example is to use several-word long segments for the analysis, which can capture the transition from one speaker to another slowly with a sliding window.

[0057] (Global and per band gain analysis) The gain determiner 212 calculates multi-band gain values, including per-band gains (adjustments) and global gains (adjustments), to be applied to the (uncorrected) speech signal based on the results generated by the short segment analyzer 230. The gain determiner 212 provides the gains to the voice enhancer 204, which applies the gains to the speech signal. The calculation of the gains can be flexible depending on the processing applied to enhance intelligibility. If there are constraints on the computing resources, the analysis bands can be grouped to effectively reduce the number of analysis bands to be processed, or some of the analysis bands can be excluded from the processing. If the processing already includes certain intelligence, such as formant location emphasis or spectral peak emphasis, the processing can use the intelligence to provide intelligibility information regarding frequency locations where the processing selectively increases / decreases the gain and appropriate global gain parameters based on the above-mentioned analysis method.

[0058] In one example, the gain may be calculated according to the following or a similar relationship: Global gain (g_Global) = Wg * St_g / Sc Gain per band (g_perband(i))=Wpb*St_pb / Sc(i) Here, g_Global and g_perband are applied to the audio output signal. Wg and Wpb are the global and per-band weighting factors. St_g and St_pb are band-wise and global short / medium length segment intelligibility values ​​(e.g., speech-to-noise (SNR) values); and Sc is the current SNR.

[0059] The weights Wg and Wpb may be determined based on a threshold intelligibility value such that the weights change relative to the current speech intelligibility value (e.g., if the intelligibility value is relatively high, then more weight (Wg) is applied to g_Global and less weight (Wpb) is applied to g_perband, and vice versa).

[0060] 9 is a high-level block / signal flow diagram of a portion of the voice enhancer 204, according to one embodiment. In the example, the voice enhancer 204 includes a multi-band compressor 904 that applies per-band gain values ​​g_pb(i) and a global gain g_Global to the speech signal to generate an intelligibility-enhanced speech signal.

[0061] 10 is a flow chart of an exemplary method 1000 for performing speech intelligibility processing, for example, performed by VIP 120. The operation of method 1000 builds on the operations described above.

[0062] At 1002, a microphone detects noise in an acoustic environment and generates a noise signal.

[0063] At 1004, an input of the VIP 120 receives an audio signal for playback into an acoustic environment via a loudspeaker.

[0064] At 1006, the VIP 120 performs a digital-to-sound level (DAL) conversion of the noise signal and performs a multi-band correction of the noise signal based on a known or derived microphone transfer function of the microphone to generate a corrected noise signal. The multi-band correction adjusts the spectrum of the noise signal to compensate for the microphone transfer function.

[0065] At 1008, the VIP 120 performs a DAL transformation of the audio signal and performs multi-band correction of the audio signal based on a known or derived loudspeaker transfer function of the loudspeaker to generate a corrected audio signal. The multi-band correction adjusts the spectrum of the audio signal to compensate for the transfer function of the loudspeaker.

[0066] At 1010, the VIP 120 determines a frequency analysis domain for the multiband speech intelligibility calculation based on the relationship between the microphone transfer function and the loudspeaker transfer function. For example, the VIP 120 determines an overlap passband where the microphone passband of the microphone transfer function and the loudspeaker passband of the loudspeaker transfer function overlap based on the start and stop frequencies of the passbands. The start and stop frequencies of a given passband can correspond to, for example, opposing 3 dB down points (or other suitable "X" dB down points) of the transfer function corresponding to the given passband.

[0067] At 1012, the VIP 120 performs a multi-band speech intelligibility analysis based on the noise signal (e.g., based on the compensated noise signal) and based on the speech signal (e.g., based on the compensated speech signal) across multiple speech analysis bands, and calculates a multi-band speech intelligibility result. The analysis can be limited to speech analysis bands of overlapping passbands, for example. The analysis result provides the speech intelligibility per band and the global speech / voice-to-noise ratio. The multi-band speech intelligibility analysis includes an analysis based on short / medium length segments / frames to obtain short-term results, and an analysis based on long segments to obtain long-term results.

[0068] At 1014, the VIP 120 calculates per-band gains and global gains based on the per-band speech intelligibility values ​​and the global speech / speech-to-noise ratio.

[0069] At 1016, the VIP enhances the intelligibility of the audio signal based on the gain and reproduces the enhanced audio signal via the loudspeaker.

[0070] In various embodiments, some of the operations of method 1000 may be omitted and / or the operations of method 1000 may be reordered / replaced. For example, transform / compensation operations 1006 and 1008 may be omitted, such that operation 1012 performs a multi-band speech intelligibility analysis based on the noise signal (uncompensated) and the speech signal (uncompensated) across multiple speech analysis bands and calculates a multi-band speech intelligibility result. In another example, operations 1006 and 1008 may be modified to omit the respective multi-band compensation, thus leaving only the respective DAL transformation.

[0071] In one embodiment, the method includes detecting noise in an environment using a microphone to generate a noise signal, receiving an audio signal to be played in the environment via a loudspeaker, determining a frequency analysis domain for multiband speech intelligibility calculation based on a relationship between a microphone transfer function of the microphone and a loudspeaker transfer function of the loudspeaker, and calculating a multiband speech intelligibility result across the frequency analysis domain based on the noise signal and the audio signal. The method further includes performing multiband correction of the noise signal based on the microphone transfer function to generate a corrected noise signal, and performing multiband correction of the audio signal based on the loudspeaker transfer function to generate a corrected audio signal, where the calculating step includes calculating a multiband speech intelligibility result across the frequency analysis domain based on the corrected noise signal and the corrected audio signal.

[0072] In another embodiment, an apparatus comprises a microphone for detecting noise in an environment and generating a noise signal, a loudspeaker for reproducing an audio signal in an underlying environment, and a controller coupled to the microphone and the loudspeaker, wherein the controller is configured to perform the steps of: performing multiband correction of the noise signal based on a microphone transfer function of the microphone to generate a corrected noise signal; performing multiband correction of the audio signal based on a loudspeaker transfer function of the loudspeaker to generate a corrected audio signal; calculating a multiband speech intelligibility result based on the corrected noise signal and the corrected audio signal; calculating multiband gain values ​​based on the multiband speech intelligibility result; and enhancing the audio signal based on the multiband gain values.

[0073] In yet another embodiment, a non-transitory computer readable medium is provided that is encoded with instructions that, when executed by a processor, cause the processor to perform the following steps: receiving a noise signal representative of noise in an environment from a microphone, receiving an audio signal to be played in the environment via a loudspeaker, performing a digital-to-sound level conversion of the noise signal and a multi-band correction of the noise signal based on a microphone transfer function to generate a corrected noise signal, performing a digital-to-sound level conversion of the audio signal and a multi-band correction of the audio signal based on a loudspeaker transfer function to generate a corrected audio signal, and calculating a multi-band speech intelligibility result including per-band speech intelligibility values ​​and a global speech-to-noise ratio based on the corrected noise signal and the corrected audio signal.

[0074] Although the technology embodied in one or more specific examples has been illustrated and described herein, it is not intended to be limited to the details shown, since various modifications and structural changes may be made within the scope of the claims and equivalents.

[0075] Each claim presented below represents a separate embodiment, and embodiments that combine different claims and / or different embodiments are within the scope of this disclosure and will be apparent to one of ordinary skill in the art after reviewing this disclosure. [Explanation of symbols]

[0076] 122 System Volume 202 Voice and Noise Analysis 204 Voice Enhancer 206 Noise Compensation Path 208 Audio correction path 210 Speech intelligibility calculation 212 Gain determiner: determines the gain adjustment value for each band and / or global gain 222 Digital Sound Level Conversion Multiband Noise Corrector for 224 Analysis 226 Digital Sound Level Conversion Multiband audio correction for 228 analysis 230 Short Segment Analysis 232 Long Segment Analysis 234 silence / pause detection 240 Pre-measured and / or calculated device / system data 242 Pre-measured and / or calculated device / system data

Claims

1. Detecting noise in an environment using a microphone to generate a noise signal; receiving an audio signal to be played in the environment via a loudspeaker; determining a frequency analysis domain for multiband speech intelligibility calculation based on a relationship between a microphone transfer function of the microphone and a loudspeaker transfer function of the loudspeaker; performing a multi-band correction of the noise signal based on the microphone transfer function to generate a corrected noise signal; performing a multiband correction of the audio signal based on the loudspeaker transfer function to generate a corrected audio signal; calculating a multi-band speech intelligibility result across the frequency analysis domain based on the corrected noise signal and the corrected speech signal; A method comprising:

2. The method of claim 1 , further comprising using the multiband speech intelligibility result to improve the intelligibility of the speech signal.

3. 2. The method of claim 1 , wherein the determining step comprises determining an overlap passband as the frequency analysis domain, where a microphone passband of the microphone transfer function and a loudspeaker passband of the loudspeaker transfer function overlap, and the calculating step comprises calculating a speech intelligibility per band over a speech analysis band limited to the overlap passband.

4. 4. The method of claim 3, wherein the calculating step further comprises calculating a global speech-to-noise ratio of (i) a voice power based on the voice signal across the speech analysis bands limited to the overlapping passbands, and (ii) a noise power based on the noise signal across the microphone passbands.

5. determining whether the start frequency of the loudspeaker passband is greater than the start frequency of the microphone passband; attenuating the audio signal to a band below a start frequency of the loudspeaker passband if the start frequency of the loudspeaker passband is above; Includes. The method according to claim 3.

6. The determining step includes: identifying start and stop frequencies that define a microphone passband and the loudspeaker passband, respectively; calculating the overlap passbands as passbands that span from a maximum of the start frequency to a minimum of the stop frequency; The method of claim 3 , comprising:

7. The method of claim 1 , wherein the step of calculating the multi-band speech intelligibility results comprises calculating per-band speech intelligibility values ​​and a global speech-to-noise ratio.

8. The step of calculating the multiband speech intelligibility result comprises: performing a multi-band speech intelligibility analysis based on short / medium length segments of the speech signal and the noise signal to generate a short-term speech intelligibility result; performing a multi-band speech intelligibility analysis based on long segments of the speech signal and the noise signal, the long segments being longer than the short / medium length segments of the speech signal and the noise signal, to generate a long-term speech intelligibility result; The method of claim 1 , comprising:

9. 1. An apparatus comprising: A microphone that detects noise in an environment and generates a noise signal; a loudspeaker for reproducing an audio signal into said environment; a controller coupled to the microphone and the loudspeaker; Equipped with The controller: performing multi-band correction of the noise signal based on a microphone transfer function of the microphone to generate a corrected noise signal; performing a multiband correction of the audio signal based on a loudspeaker transfer function of the loudspeaker to generate a corrected audio signal; calculating a multiband speech intelligibility result based on the corrected noise signal and the corrected speech signal; enhancing the intelligibility of the speech signal using the multiband speech intelligibility result; The apparatus is configured to:

10. In the step of improving the intelligibility of the speech signal using the multiband speech intelligibility result, the controller calculating multiband gain values ​​based on the multiband speech intelligibility results; enhancing the audio signal based on the multiband gain values; The apparatus of claim 9 , further configured to:

11. The controller: determining an overlap passband where a microphone passband of the microphone transfer function and a loudspeaker passband of the loudspeaker transfer function overlap; 10. The apparatus of claim 9, wherein the controller is configured to perform the calculation by calculating a per-band speech intelligibility value over speech analysis bands limited to the overlapping passbands.

12. 10. The apparatus of claim 9, wherein the controller is configured to perform the calculation of the multi-band speech intelligibility results by calculating per-band speech intelligibility values ​​and a global speech-to-noise ratio.

13. 4. The method of claim 3, further comprising: performing a multi-band speech intelligibility analysis on short / medium length segments of the corrected speech signal and the corrected noise signal to generate short term speech intelligibility results; performing a multiband speech intelligibility analysis on long segments of the corrected speech signal and the corrected noise signal, the long segments being longer than the short / medium length segments of the corrected speech signal and the corrected noise signal, to generate a long term speech intelligibility result; The apparatus of claim 9 , comprising:

14. performing a digital-to-sound level conversion of the noise signal based on the sensitivity of the microphones prior to multiband correction of the noise signal; performing a digital-to-sound level conversion of the audio signal prior to multiband correction of the audio signal; The apparatus of claim 9 further comprising:

15. When executed by the processor, receiving a noise signal representative of noise in an environment from a microphone; receiving an audio signal to be played in the environment via a loudspeaker; performing a digital-to-sound level conversion of the noise signal and a multi-band correction of the noise signal based on a microphone transfer function to generate a corrected noise signal; performing a digital-to-sound level conversion of the audio signal and a multi-band correction of the audio signal based on a loudspeaker transfer function to generate a corrected audio signal; calculating a multi-band speech intelligibility result based on the corrected noise signal and the corrected speech signal, the multi-band speech intelligibility result including a per-band speech intelligibility value and a global speech-to-noise ratio; A non-transitory computer-readable medium encoded with instructions to cause the processor to execute.

Images